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This  publication  Is  prlmrlly  a  working  paper.  It  Is  published  solely  to  documnt  work  performed 


The  Air  Force  Human  Resources  Laboratory  is  currently  engaged  in  a  high-resource  job  perform¬ 
ance  criterion  development  effort.  Work  sample  tests  and  rating  forms  have  been  developed  to 
measure  the  job  proficiency  of  first-term  jet  engine  mechanics.  A  vital  component  of  this 
measurement  process  centers  on  the  administrator's  ability  to  accurately  rate  job  incumbents.  A 
necessary  part  of  this  research  and  development  effort  is  the  training  of  both  work  sample  and 
rating  form  evaluators.  Therefore,  a  training  program  was  developed  and  administered  to 
evaluators.  The  product  of  this  work  is  a  trainer's  manual  which  provides  the  details  for 
conducting  a  systematic  program  of  test  administrator  training.  The  manual  includes  discussions 
of  work-sample  test  administration  procedures  and  requirements;  hands-on,  interview,  and  rater 
training  guidelines;  and  long-term  training  recommendations. 


PREFACE 


Planning  for  the  Air  Force's  program  of  research  in  performance  assessment  began 
several  years  ago  as  the  result  of  three  primary  requirements.  Operational  military  and 
civilian  program  managers  In  the  Manpower,  Personnel,  and  Training  comminltles  asked  the 
Air  Force  Human  Resources  Laboratory  (AFHRL)  to  develop  a  technology  for  measuring  job 
performance  so  that  the  measures  could  be  used  to  evaluate  the  effectiveness  of  training 
and  selection  programs  and  effects  of  other  personnel  programs  and  procedures.  The 
Uniform  Guidelines  for  Employee  Selection  of  1978,  as  supported  by  a  review  of  case  law. 
Indicate  the  civilian  procedures  must  be  validated  against  job  performance  criteria  to 
avoid  litigation. 

Second,  the  Manpower,  Personnel,  and  Training  research  community  neeoed  the 
performance  measures  to  serve  as  criteria  In  their  many  RIO  projects.  An  AFHRL  Research 
Advisory  Panel  recommended  that  the  many  RIO  efforts  to  obtain  specific  performance 
criteria  be  consolidated  Into  one  combined  effort.  Thus,  a  job  performance  measurement 
technology  composed  of  a  wide  range  of  candidate  measures  could  be  developed  to  serve 
these  varied  needs. 

Plans  for  the  Air  Force  performance  measurement  effort  to  meet  these  two 
requirements  were  under  development  when  a  third  requirement  for  the  measures  came  with 
a  Congressional  mandate  that  military  selection  tests  be  validated  against  job 
performance  measures.  These  operational  requirements,  and  legal  and  Congressional 
mandates  provided  the  impetus  to  planning  and  obtaining  support  for  this  lengthy 
high-resource  RIO  effort. 

The  Air  Force  is  Involved  In  a  long-term  RIO  effort  to  systematically  obtain  job 
performance  measures.  The  short-term  objective  of  this  effort  Is  the  development  of 
on-the-job  performance  measures  to  validate  Air  Force  selection  and  classification 
procedures.  Guidelines  for  developing  and  obtaining  the  performance  measures  will  be 
established  for  a  wide  range  of  enlisted,  officer,  and  civilian  jobs.  Once  obtained, 
these  measures  will  be  placed  in  a  data  base  for  validation  use.  The  long-term  goal  is 
to  establish  an  operational  performance  measurement  program  for  evaluation  of  selection 
and  training  procedures  and  personnel  policies  and  practices.  The  goal  here  Is  to 
operationalize  the  procedures  so  that  the  performance  measurement,  validation,  and 
evaluation  can  be  carried  out  by  technicians,  as  Is  currently  done  by  the  USAF 
Occupational  Measurement  Center  with  the  Occupational  Survey  (Job  Analysis)  Program. 

The  Air  Force  RIO  plan  has  not  been  developed  In  Isolation  from  the  other 
services.  The  Air  Force  Is  Involved  with  the  Army,  Navy,  and  Marine  Corps  In 
coordinating  the  Congressional 1y  mandated  Joint-Service  Job  Performance  Measurement 
Project  to  link  enlistment  standards  to  Job  performance.  The  Air  Force's  major  focus  in 
this  effort  Is  a  new  work  swple  testing  technique  called  Walk-Through  Performance 
Testing  (WTPT).  It  combines  testing  of  hands-on  performance  with  Interview  testing  to 
provide  a  high-fidelity  measure  of  Individual  technical  job  competence.  In  addition, 
supervisory,  peer,  and  self-rating  forms  have  been  developed  at  several  levels  of 
specificity.  Finally,  several  questionnaires  have  also  been  developed  to  assess  factors 
related  to  performance  measurement  (FRPM)  such  as  Job  experience  and  motivation. 

In  order  to  collect  the  most  accurate  data  from  test  administrators,  supervisors, 
peers,  and  Incwbents,  structured  training  progrims  are  required.  The  purpose  of  this 
trainer's  manual  Is  to  provide  the  background  and  training  materials  necessary  to 
accomplish  the  training  function. 
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JOS  PERFORMANCE  MEASUREMENT  SYSTEM 
TRAINER'S  MANUAL 


I.  INTRODUCTION 

The  chief  requirement  of  the  Air  Force  criterion  development  project  is  to  ensure  that  the 
most  accurate  job  performance  data  are  collected.  This  means  that  at  every  stage  of  the  process 
from  development  through  data  collection,  steps  must  be  taken  that  increase  the  chances  that 
accurate  data  will  be  obtained.  A  critical  part  of  this  strategy  is  the  training  of  both  test 
administrators  and  raters. 

i 

The  purpose  of  this  trainer's  manual  is  to  explicate  the  background,  procedures,  and 
techniques  necessary  to  collect  accurate  data  using  the  instruments  in  the  Job  Performance 
Measurement  System  (JPMS);  i.e.,  Walk-Through  Performance  Testing  (WTPT),  rating  forms,  and 
Factors  Related  to  Performance  Measurement  Questionnaires.  To  accomplish  these  objectives  for 
the  Jet  Engine  Mechanic  Specialty  (AFS  426X2)  and  to  serve  as  a  model  for  future  Air  Force 
specialties  (AFSs),  information  is  included  on  both  preworkshop  training  activities  and  a  formal 
training  workshop  held  prior  to  data  collection.  Because  the  goal  of  training  is  to  make  test 
administrators  and  raters  as  reliable,  valid,  and  accurate  in  their  ratings  as  possible,  each 
training  component  has  been  designed  with  that  end  in  mind. 

The  first  major  section  describes  preworkshop  training  activities  and  recommendations  for 
their  use  prior  to  the  formal  training  workshop.  A  second  component  focuses  on  the  training 
workshop  Itself,  with  major  sections  on  WTPT  administration,  interview  training,  a  behavior 
modeling  exercise,  hands-on  training,  and  rater  training.  The  next  section  includes  sets  of 
scripts  and  slides  useful  in  training  the  test  administrators  and  raters,  as  well  as  scripts  and 
slides  to  be  used  by  the  test  administrators  to  train  raters  in  the  field.  An  interview  modeling 
exercise  demonstrates  proper  conduct  of  an  interview  test  item,  thus  providing  an  opportunity  for 
test  administrators  to  observe  the  modeling  of  appropriate  Interview  behavior.  One  additional 
section  discusses  some  variations  and  alternatives  to  the  general  training  plan  presented  in  this 
manual.  It  discusses  issues  such  as  (a)  use  of  active-duty  versus  contractor-hired  test 
administrators,  (b)  length'  of  time  allotted  for  training,  (c)  use  of  test  proctors,  and  (d) 
shadow  scoring/recallbratlon.  The  final  section  provides  concluding  remarks. 

This  training  manual  has  been  organized  and  assembled  to  provide  the  best  training  possible 
for  test  administrators  and  raters.  In  addition,  it  has  been  constructed  in  a  way  that  should 
provide  the  trainer  with  a  readily  accessible,  easy-to-follow  set  of  materials.  Whenever 
AFS-specIf 1c  training  materials  are  required,  the  manual  utilizes  Jet  engine  mechanic  information 
as  examples  for  other  specialties.  It  is  hoped  the  training  manual  will  provide  a  prototype  for 
future  specialties. 


II.  PREMORKSHOP  ACTIVITIES 

If  the  trainer  is  fortunate  enough  to  be  granted  an  extended  period  of  time  with  the  test 
administrators  prior  to  a  structured  workshop,  certain  training  activities  should  be  undertaken 
that  may  prove  Invaluable  to  the  success  of  the  overall  training  effort.  It  should  be  noted  that 
active-duty  test  administrators  will  not  be  available  for  extended  periods  prior  to  data 
collection.  Therefore,  the  coements  that  follow  are  directed  at  contractor-hired  test 
administrators. 
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Orientation  Briefing 

Initially,  all  WTPT  administrators  participate  in  an  orientation  briefing.  This  meeting 
occurs  as  soon  as  all  administrators  are  hired,  and  lasts  2  or  3  hours.  The  purpose  of  the 
meeting  is  to  familiarize  the  administrators  with  the  project  in  general,  and  to  introduce  to  the 
administrators  key  personnel  Involved  In  the  endeavor.  Most  Important,  the  orientation 
thoroughly  outlines  the  responsibilities  of  the  administrators  and  emphasizes  the  criticality  of 
their  roles  in  the  research  project.  In  addition,  a  calendar  of  events  is  distributed  to  all 
administrators  indicating  milestones  to  be  reached  in  the  months  that  follow.  These  milestones 
are  discussed  to  ensure  that  the  administrators  understand  what  is  expected  of  them  and  when.  A 
suggested  agenda  is  included  in  Appendix  A. 


Hands-On  Tralnlno 

Although  the  administrators  are  experts  In  the  field  In  which  they  gather  data,  the  accuracy 
of  their  observations  is  increased  if  they  become  technically  proficient  at  performing  each  task 
included  in  WTPT.  The  technical  proficiency  is  achieved  over  a  period  of  several  weeks.  The 
administrators  are  taken  to  a  base  for  observation  of  active-duty  incumbents  performing  the  tasks 
included  In  the  WTPT.  The  objective  of  this  training  Is  twofold.  First,  the  test  administrators 
gain  valuable  practice  observing  correct  and  incorrect  performance.  In  addition,  the 
administrators  can  use  WTPT  booklets  to  record  observations,  thus  becoming  more  familiar  with 
using  the  booklets,  and  focusing  on  key  steps  in  the  WTPT  tasks. 

During  this  period,  test  administrators  are  given  the  opportunity  to  gain  hands-on  experience 
performing  WTPT  tasks.  Also,  by  this  time,  the  trainer  should  know  whether  equipment  required  to 
perform  tasks  In  the  WTPT  Is  available  for  the  formal  training  workshop.  Given  equipment 
availability,  the  trainer  may  choose  to  have  each  administrator  perform  certain  WTPT  tasks  in  the 
workshop  for  all  other  administrators  to  observe  and  score.  This  strategy  provides  a  "live" 
hands-on  training  exercise  that  can  prove  quite  valuable,  especially  If  the  "performing"  test 
administrator  Is  directed  to  make  certain  errors  while  doing  the  task. 

If  equipment  is  not  available  at  the  workshop,  then  an  alternative  to  "live"  hands-on 
training  Is  required.  Videotaping  provides  a  very  useful  alternative.  Either  while  In  the  field 
for  this  preworkshop  hands-on  training,  or  during  the  final  stages  of  WTPT  development, 
videotapes  of  active-duty  subject-matter  experts  (SMEs)  performing  WTPT  tasks  can  be  made.  Much 
planning  Is  required  to  make  the  videotape  endeavor  worthwhile.  Performance  should  be  scripted, 
with  SME  Input  as  to  where  logical  (and/or  subtle)  errors  can  be  made.  Also,  it  Is  advisable  to 
use  experienced  noncommissioned  officers  (NCOs)  as  actors  to  ensure  precisely  accurate  or 
Inaccurate  performances.  Finally,  whether  videotape  or  "live"  hands-on  performances  are  used, 
any  time  an  Incorrect  performance  Is  shown.  It  should  be  followed  by  a  correct  performance  to 
reinforce  the  desired  behaviors. 

Videotaping  can  be  quite  a  useful  training  tool  because  with  careful  scripting  and 
performing,  the  trainer  subsequently  possesses  knowledge  of  the  target  score  (which  steps  were 
performed  correctly/which  were  not)  for  each  task,  and  can  therefore  judge  whether  the  test 
administrators  are  scoring  accurately  or  not.  More  will  be  discussed  about  the  use  of  videotape 
technology  for  training  In  a  later  section. 

A  good  deal  of  precoordination  and  planning  1$  required  to  successfully  Implement  the 
hands-on  training  process.  The  entire  hands-on  training  process  required  substantial  travel  by 
the  administrators  involved  In  the  det  Engine  Mechanic  Specialty  effort,  since  only  one  of  the 
three  engines  under  consideration  was  locally  available.  In  addition,  the  videotaping  component 
necessitated  support  by  trained  technicians. 
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Some  of  the  most  important  training  that  takes  place  prior  to  the  formal  workshop  is  not  a 
structured  activity  or  exercise  but  rather,  a  continuing  process.  It  involves  substantial 
reading  and  stuaying  of  tests,  the  administrator's  manual,  relevant  technical  orders,  rating 
forms,  and  rater  training  and  background  information  on  the  research  effort.  This  long-term 
training  begins  for  the  administrator  immediately  upon  hiring  and  continues  until  data  collection. 

A  significant  part  of  the  process  includes  role-play  between  administrators,  with  one  playing 
the  role  of  an  incumbent  and  the  other,  the  administrator.  Although  most  of  the  hands-on  tests 
in  a  technical  field  such  as  the  Jet  Engine  Mechanic  Specialty  cannot  be  effectively  role-played 
in  an  office  environment,  all  administrators  can  certainly  achieve  proficiency  on  the  interview 
tests.  Role-play  of  interview  tests  ensures  that  the  administrator  can  be  understood  by  an 
Incumbent  and  also  allows  the  administrator  to  practice  handling  an  array  of  verbal  responses  by 
an  Incumbent.  Role-playing  also  provides  practice  in  greeting  an  incumbent,  beginning  a  day  of 
testing,  making  transitions  between  tests,  and  closing  a  day  of  testing. 

Another  component  of  the  ongoing  training  process  Involves  the  administrators'  establishment 
of  several  testing  schedules  (for  sequencing  of  tasks  and  examinees)  to  be  used  during  data 
collection.  While  working  on  the  schedules,  the  administrators  become  more  familiar  with  each 
test,  specifically  the  established  time  limits  of  each  test,  and  the  location  of  each  test 
activity. 

Finally,  the  administrators  may  be  used  to  assist  in  the  establishment  of  scoring  procedures 
for  the  tests,  especially  in  the  area  of  weighting  the  test  items.  Again,  this  is  another  means 
of  familiarizing  the  administrators  with  the  tests. 


III.  TRAINING  WORKSHOP 

This  workshop  has  been  designed  to  cover  the  fundamental  requirements  for  Walk-Through 
Performance  Testing  (WTPT)  and  rating  form  administration.  Key  elements  in  this  process  are 
emphasized  through  lecture,  discussion,  observation,  practice,  and/or  feedback.  The  training 
workshop  benefits  both  the  administrators  and  the  trainer.  For  the  administrators,  it  is  a 
culmination  of  all  training  received  In  previous  weeks  and  provides  a  chance  for  them  to 
-ssimilate  the  information  and  thus  be  able  to  apply  it  effectively.  In  those  instances  where 
training  time  is  extremely  limited  (l.e.,  active-duty  test  administrators  are  used,  and  can  be 
removed  from  their  assigned  duty  station  for  only  a  limited  period  of  time),  the  training 
workshop  may  be  the  only  training  test  administrators  receive.  The  workshop  also  permits  the 
trainer  to  evaluate  the  capabilities  of  the  administrators  and  identify  any  weaknesses  in  their 
skills. 

The  workshop  should  be  scheduled  no  more  than  2  or  3  weeks  prior  to  the  administration  of  the 
WTPT  and  rating  forms.  This  time  interval  Is  short  enough  to  allow  training  material  to  remain 
fresh  in  the  minds  of  the  administrators,  but  gives  the  trainer  adequate  time  to  provide  extra 
attention  to  those  administrators  who  need  further  training  or  practice  in  certain  areas. 

The  workshop  should  be  conducted  in  a  quiet  room,  large  enough  to  comfortably  accommodate  all 
administrators.  The  room  should  be  equipped  with  tables  or  desks  and  an  overhead  projector.  The 
Interview  Training  component  of  the  workshop  requires  the  use  of  a  videocassette  player.  The 
Hands-On  Training  component  requires  several  additional  rooms  (and  If  videotapes  are  used,  a 
videocassette  player  in  each  room).  Appendix  B  contains  a  checklist  to  facilitate  preparation 
for  the  workshop. 


The  workshop  is  broken  into  several  major  sections.  The  first  item  to  be  addressed  is  the 
agenda  to  oe  followed  during  the  workshop.  Next,  an  overview  briefing  is  presented  to  provide 
the  administrators  with  extensive  information  about  project  background  and  current  developmental 
efforts. 

Following  the  overview  briefing,  logistical  requirements  associated  with  visits  to  the  data 
collection  oases  should  be  discussed.  One  potential  responsibility  of  the  administrators  upon 
arriving  at  each  base  is  to  deliver  an  in-briefing  to  the  senior  base  representative  for  the 
specialty  (e.g.,  for  Jet  Engine  Mechanic,  this  would  be  the  Deputy  Commander  for  Maintenance 
(DCM) ) .  If  a  test  proctor  is  assigned  to  accompany  the  test  administrator  team,  briefing 
responsibilities  may  be  given  to  the  proctor.  Nevertheless,  this  briefing  should  be  given  to  the 
administrators  and  then  time  allowed  for  them  to  practice  briefing  each  other. 

The  bulk  of  the  workshop  concentrates  on  learning  and  mastering  skills  necessary  to 
administer  the  WTPT  effectively.  This  area  is  divided  into  three  sections:  a  discussion  of  test 
administration  requirements;  discussion,  modeling,  and  practice  of  good  interviewing  techniques; 
and  practice  using  the  hands-on  tests  of  WTPT. 


WTPT  Training 

For  the  Air  Force,  hands-on  testing  is  a  particular  problem  because  of  the  complexity  and 
expense  involved  in  performing  many  tasks.  For  example,  many  critical  tasks  cannot  be  measured 
by  hands-on  testing  because  these  tasks  tend  to  take  too  long  to  complete,  require  replacement  of 
expensive  parts,  and  risk  personal  Injury  or  damage  to  components.  The  AFHRL  has  devised  a  new 
methodology  to  deal  with  these  problems.  This  new  approach,  Walk-Through  Performance  Testing 
(WTPT),  has  as  Its  foundation  the  work-sample  philosophy,  but  attempts  to  expand  through  the  use 
of  interview  testing  the  measurement  of  critical  tasks  not  measured  by  hands-on  testing. 

Interview  testing  requires  the  administrator  to  assess  an  incumbent's  proficiency  on  a  task 
by  asking  questions  designed  to  uncover  prof iciency- based  strengths  and  weaknesses  related  to  the 
performance  of  that  task.  The  interview  testing  Is  conducted  at  the  worksite  in  a 
"show-and-tell"  fashion  so  that  the  person  being  evaluated  can  visually  and  verbally  describe  how 
a  step  is  to  be  accomplished  (e.g.,  "that  bolt  is  to  be  turned  five  revolutions,"  or  "that 
component  is  to  be  lubricated  prior  to  being  assembled").  Thus,  additional  information,  not 
otherwise  collected,  can  be  assembled,  along  with  hands-on  information,  to  provide  a  more 
thorough  coverage  of  the  content  domain,  and  hopefully,  a  more  accurate  picture  of  an 
individual's  job  proficiency. 

Because  of  this  measurement  orientation,  the  focus  of  WTPT  training  will  be  threefold. 
Initial  discussions  will  center  on  procedural  issues,  such  as  how  to  administer  and  score  WTPT 
work  samples.  Here,  much  of  the  discussion  can  focus  on  the  Administrator's  Manual  developed  by 
Alba  and  Wilcox  (1985).  A  second  component  of  training  is  interview  skills  development.  Because 
of  the  interview  component  of  the  WTPT,  test  administrators  will  need  to  become  familiar  with 
conducting  interviews  and  using  effective  probes.  A  third  section  will  concentrate  on  scoring 
hands-on  tasks.  The  administrators  will  observe  videotapes  of  tasks  being  performed  and  rate 
each  individual's  performance  in  a  test  booklet. 


Test  Administration  Requirements 


To  organize  and  coordinate  test  administration  and  data  collection,  standardized  procedures 
need  to  be  developed  and  subsequently  explained  in  detail  to  test  administrators.  During  this 
phase  of  the  workshop,  the  administrators  will  be  trained  in  the  standardized  testing  procedures 


outlined  In  the  WTPT  Administrator's  Manual.  Allow  several  hours  to  accomplish  this.  Copies  of 
the  manual  are  included  in  the  administrators'  packets  of  workshop  materials.  If  the 
administrators  have  not  been  exposed  to  the  manual  previously,  allow  sufficient  time  for  them  to 
survey  it  briefly.  Then,  read  through  the  manual  with  the  group,  section  by  section.  It  is 
critical  that  test  administrators  have  a  thorough  knowledge  ana  understanding  of  the  testing 
procedures,  to  allow  them  to  elaborate  on  critical  areas  or  resolve  any  issues  that  might  arise. 
Emphasize  to  the  administrators  that  reliability  of  the  test  data  they  collect  is  dependent  upon 
their  adherence  to  the  standardized  testing  procedures.  The  administrators  must  know  ana 
understand  the  information  contained  in  the  manual  and  be  able  to  perform  their  duties 
competently  and  without  hesitation.  Instruct  the  administrators  that  it  is  necessary  to  spend 
considerable  time  outside  of  the  workshop  studying  the  manual  in  order  to  reach  this  level. 


Interview  Training 

As  noted  earlier,  interview  testing  is  an  important  component  of  the  WTPT  process.  This 
approach  to  performance  measurement  requires  that  administrators  receive  interview  skills 
training.  The  interview  situation  is  a  face-to-face  Interaction  between  the  administrator  and 
incumbent;  thus,  the  administrator  must  possess  special  skills  that  enable  him/her  to  achieve  the 
required  pattern  of  interaction.  What  is  demanded  of  the  interviewer  is  the  learning  of  a 
specialized  way  of  behaving  and  interacting  with  another  individual  for  the  purpose  of  improving 
communication.  This  means  acquiring  additional  insight  into  the  interactions  that  make  up  the 
interview,  as  well  as  understanding  more  about  the  forces  that  motivate  the  respondent  and 
influence  the  administrator's  reaction  to  the  examinee. 

Because  of  the  interview  testing  component  of  WTPT,  an  Important  part  of  this  training  must 
focus  on  strengthening  administrator  Interviewing  skills.  This  training  will  focus  first  on  a 
lecture/discussion  of  proper  Interviewing  techniques.  It  should  then  be  followed  by  a  behavior 
modeling  exercise  which  Includes  (a)  behavioral  modeling,  (b)  rehearsal,  and  (c)  feedback. 


Modeling  Exercise 

Following  the  lecture/group  discussion,  proper  conduct  of  an  interview  test  item  will  be 
demonstrated  via  a  behavior  modeling  exercise.  The  exercise  requires  two  actors  to  role-play  an 
interview  test  item  and  utilize  appropriate  communication  and  probing  techniques  throughout  the 
interview.  It  is  necessary  for  the  two  actors  to  rehearse  the  item  until  they  can  effectively 
demonstrate  the  desired  interview  techniques.  The  actors  should  also  demonstrate  appropriate 
methods  of  opening  and  closing  the  Interview.  This  is  not  to  be  a  participatory  activity  for 
administrators  but  rather,  an  opportunity  for  them  to  observe  the  modeling  of  appropriate 
interview  behavior.  A  discussion  follows  the  observation  of  the  interview,  in  which  the  various 
techniques  used  by  the  actors  are  highlighted  and  discussed.  Subsequent  to  the  modeling 
exercise,  the  administrators  will  have  a  chance  to  practice  proper  interview  procedures  by  role- 
playing  interview  test  items  with  each  other. 

Section  IV  contains  a  script  used  for  the  behavior  modeling  exercise  during  the  training 
workshop  for  administrators  of  the  WTPT  in  the  Jet  Engine  Mechanic  Specialty.  Interview  item 
#134  was  chosen  for  the  exercise  because  it  Is  lengthy  enough  to  allow  the  demonstration  of  a 
variety  of  interview  techniques.  The  Interview,  including  opening  and  closing,  was  rehearsed 
extensively  and  then  videotaped  prior  to  the  workshop.  Videotaping  allows  for  standardization  of 
the  behavior  being  modeled.  This  Is  Important  because  the  discussion  component  of  the  exercise 
is  greatly  facilitated  by  replaying  the  segments  of  the  tape  that  illustrate  the  various 
techniques.  In  addition,  the  videotape  can  be  replayed  at  a  later  date  if  a  follow-up  workshop 
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Is  necessary.  It  is  strongly  recommended  that  the  videotape  approach  be  used  by  trainers  of 
administrators  in  other  specialties. 

Hands-On  Training 

In  this  third  segment  of  WTPT  training,  test  administrator  teams  are  shown  correct  procedures 
for  performing  all  WTPT  hands-on  tasks.  The  aim  in  these  exercises  is  to  calibrate  the 
observation  and  rating  processes  of  the  administrators.  The  time  required  for  this  training 
component  may  vary  due  to  the  experience  of  the  administrators.  It  is  imperative  that 
administrators  are  rating  WTPT  in  the  same  way. 

If  videotapes  are  to  be  shown  for  WTPT  hands-on  tasks,  administrators  are  required  to  rate  an 
Individual's  performance  on  those  tasks.  A  task  is  shown  being  performed  either  correctly  or 
Incorrectly.  After  each  task  is  performed  and  evaluated  in  WTPT  booklets,  the  group  should  reach 
consensus  on  their  ratings  of  each  step  of  the  task,  as  well  as  their  overall  rating  of  the 
task.  On  those  tasks  that  were  shown  being  performed  Incorrectly,  the  correct  procedures  are 
shown  following  the  group  discussion. 

If  the  WTPT  development  process  has  generated  Phase  II  or  Phase  III  tasks,  the  large  group 
will  need  to  separate  Into  their  evaluation  teams.  Separate  rooms  and  videocassette  players  need 
to  be  reserved  beforehand  for  this  exercise. 


Rater  Training 

Another  major  component  of  the  workshop  requires  the  administration  of  a  Rater  Training 
Session  to  the  administrators.  This  duty  may  also  be  assigned  to  the  test  proctor.  The  training 
Is  Identical  to  that  which  administrators  may  present  at  each  base  prior  to  administration  of  the 
rating  forms. 

Although  the  administrators  may  already  be  familiar  with  the  content  of  the  rater  training, 
this  Is  an  opportunity  for  them  to  view  a  demonstration  of  the  training,  and  to  experience  it 
from  an  Incumbent's  viewpoint.  Rater  training  materials  are  included  In  Appendices  C  -  F. 

The  workshop  concludes  with  sumnary  remarks  and  distribution  of  the  administrative  checklist 
which  highlights  responsibilities  of  administrators  at  each  base.  All  training  materials  needed 
for  this  workshop  are  found  In  Section  IV  or  in  the  appendices. 


IV.  TRAINING  WORKSHOP  SCRIPTS  AND  SLIDES 

This  section  contains  all  scripts  and  slides  necessary  for  conducting  the  training  workshop. 
While  most  slides  are  generic  and  can  be  used  for  any  AFS,  It  is  necessary  In  some  Instances  to 
address  a  specific  career  field.  In  these  cases,  AFS  426X2  Is  used  as  an  example.  Those  slides 
requiring  AFS-specIflc  Information  are  noted,  as  necessary.  In  the  narrative  preceding  each  set 
of  slides. 

The  Workshop  Introduction  portion  of  this  section  consists  of  two  slides  and  an  accompanying 
script  to  be  used  by  the  trainer  to  open  the  workshop. 
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Workshop  Introduction  Script 
SHOE  »1  -  ADMINISTRATOR  TRAINING  WORKSHOP 

HELLO,  MY  NAME  IS _ .  I‘0  LIKE  TO  WELCOME  YOU  TO  THIS 

WORKSHOP.  OUR  GOAL  WILL  8E  TO  FAMILIARIZE  YOU  WITH  ALL  THE  MATERIALS  AND  INFORMATION  YOU  WILL 
NEED  TO  AOMINISTER  THE  WALK-THROUGH  PERFORMANCE  TEST  (WTPT),  RATING  FORMS,  AND  RELATED 
QUESTIONNAIRES.  IN  ADDITION,  THE  WORKSHOP  HAS  BEEN  DESIGNED  TO  STRENGTHEN  THE  ACCURACY  ANO 
CONSISTENCY  OF  PERFORMANCE  OBSERVATION  IN  THE  WTPT,  AS  WELL  AS  PROCEDURAL  CONSISTENCY  ACROSS 
ADMINISTRATORS. 

AT  THIS  POINT,  I'D  LIKE  TO  HAND  OUT  A  PACKAGE  OF  MATERIALS  THAT  YOU  WILL  USE  THROUGHOUT  THE 
WORKSHOP.  (Note  to  Trainer:  see  Appendix  B  for  checklist  of  packet  materials.) 

SLIDE  #2  -  SCHEDULE 

THE  WORKSHOP  IS  DIVIOEO  INTO  FIVE  MAIN  COMPONENTS: 

1.  BACKGROUND/OVERVIEW 

2.  LOGISTICAL  REQUIREMENTS 

3.  BASE  INTRODUCTORY  BRIEFING 

4.  WTPT  TRAINING 

5.  RATING  FORM  TRAINING 


FIRST,  I  WILL  PRESENT  A  FAIRLY  OETAILEO  COVERAGE  OF  THE  BACKGROUND,  CURRENT  DEVELOPMENTAL 
WORK,  AND  RELATED  EFFORTS  RELEVANT  TO  THIS  PROJECT.  I  WANT  TO  PROVIDE  YOU  WITH  SOME  OF  THE 
REASONS  WHY  THIS  PROJECT  HAS  BEEN  UNDERTAKEN  BEFORE  TURNING  OUR  FOCUS  TO  DATA  COLLECTION  ISSUES. 

AFTER  THIS  OVERVIEW  BRIEFING,  LOGISTICAL  REQUIREMENTS  OF  THE  WORK  WILL  BE  DISCUSSED.  HERE  WE 
WILL  BE  TALKING  ABOUT  SCHEDULING  OF  ACTIVITIES  SUCH  AS  RATER  TRAINING,  ADMINISTRATION  OF  RATING 
FORMS,  ANO  ENSURING  CORRECT  CONFIGURATION  OF  ENGINES.  UPON  YOUR  ARRIVAL  AT  THE  DATA  COLLECTION 
SITE,  ONE  OF  THE  FIRST  REQUIREMENTS  WILL  BE  TO  BRIEF  KEY  AIR  FORCE  PERSONNEL  ON  THE  REASONS  FOR 
YOUR  PRESENCE,  WHAT  THE  TESTING  SCHEDULE  IS  EXPECTED  TO  BE,  AND  WHAT  ASSISTANCE  YOU  WILL  NEED 
FROM  THEM.  CONSEQUENTLY,  WE  HAVE  PREPARED  A  SHORT  PRESENTATION  THAT  CAN  SERVE  AS  A  MODEL  FOR 
THIS  BRIEFING.  FOR  THE  JET  ENGINE  MECHANIC  SPECIALTY,  YOU  CAN  EXPECT  THIS  BRIEFING  TO  BE 
ATTENDED  BY  THE  DEPUTY  COMMANDER  FOR  MAINTENANCE  (DCM)  AND  THE  WING  COMMANDER.  I  WILL  PRESENT 
THIS  BRIEFING  TO  YOU,  ANO  THEN  ALL  OF  YOU  WILL  BE  GIVEN  TIME  TO  PRACTICE  THE  BRIEFING. 

FOLLOWING  THE  BASE  INTRODUCTORY  BRIEFING,  WE  WILL  DISCUSS  THE  WTPT  ADMINISTRATION 
REQUIREMENTS.  THE  WTPT  ADMINISTRATOR'S  MANUAL  WILL  BE  THE  PRIMARY  GUIDE  FOR  THIS  TRAINING. 

NEXT  WE  WILL  FOCUS  ON  THE  INTERVIEW  TESTING  COMPONENT  OF  WTPT.  WE  WILL  DISCUSS  INTERVIEWING 
SKILLS,  VIEW  A  VIOEOTAPE  OF  HOW  TO  CONOUCT  A  PROPER  INTERVIEW,  AND  PRACTICE  THESE  SKILLS. 

FOLLOWING  YOUR  INTERVIEW  TRAINING,  WE  WILL  FOCUS  ON  THE  HANDS-ON  TESTING  COMPONENT  OF  WTPT. 
VIDEOTAPES  OF  JET  ENGINE  MECHANICS  PERFORMING  WTPT  HANDS-ON  TASKS  WILL  BE  SHOWN,  ANO  YOU  WILL  BE 
REQUIRED  TO  RATE  THE  INDIVIDUAL'S  PERFORMANCE  ON  EACH  TASK.  RATINGS  WILL  THEN  BE  COMPARED  AND 
DISCUSSEO. 

THE  FINAL  SESSION  OF  THE  WORKSHOP  WILL  FOCUS  ON  RATER  TRAINING  AND  RATING  FORM 
ADMINISTRATION.  I  WILL  PRESENT  THE  RATER  TRAINING  SESSION  TO  YOU  JUST  AS  YOU  WILL  HAVE  TO 
PRESENT  IT  TO  THE  SUPERVISORS,  COWORKERS,  ANO  INCUMBENTS. 


Workshop  Introduction  Slides 


RATING  FORM  TRAINING 


Overview  Briefing  Script 

Prior  to  delivering  this  briefing  to  workshop  attendants,  the  trainer  should  set  the  stage  by 
explaining  that  the  Intent  of  the  briefing  Is  to  convey  very  detailed  Information  about  the  Air 
Force  Job  Performance  Measurement  Project.  During  data  collection,  the  test  administrators  are 
field  representatives  of  the  project  and  can  expect  to  be  asked  questions  about  the  project  by 
various  base  personnel.  This  briefing  supplies  most  of  the  Information  needed  to  answer  such 
questions. 


SLIDE  »1  -  OVERVIEW 

IN  THIS  BRIEFING,  I'LL  EXPLAIN  THE  OBJECTIVES  OF  OUR  JOB  PERFORMANCE  MEASUREMENT  WORK,  THE 
REQUIREMENTS  THAT  ARE  DRIVING  THE  WORK,  OUR  ASSOCIATION  WITH  A  LARGER  JOINT-SERVICE  EFFORT,  AND 
THE  OETAILS  OF  THE  AIR  FORCE  EFFORT. 


SLIDE  #2  -  OBJECTIVES 

WE  ARE  DEVELOPING  A  JOB  PERFORMANCE  MEASUREMENT  TECHNOLOGY  TO  VALIDATE  THE  AIR  FORCE'S 
SELECTION  AND  CLASSIFICATION  PROCEDURES  AND,  MORE  SPECIFICALLY,  TO  TEST  THE  FEASIBILITY  OF  USING 
THESE  MEASURES  TO  SET  ENLISTMENT  STANDARDS.  IN  ADDITION,  THIS  INFORMATION  CAN  BE  USED  TO 
EVALUATE  AIR  FORCE  MANPOWER,  PERSONNEL,  AND  TRAINING  SYSTEMS.  FINALLY,  WE  HOPE  TO  USE  THIS 
INFORMATION  TO  ACHIEVE  COMPATIBLE  INDIVIDUAL  AND  UNIT  PERFORMANCE  MEASUREMENT  SYSTEMS. 


SLIDE  *3  -  SELECTION  SYSTEM  VALIDATION  CYCLE 

IN  THE  PAST,  THE  AIR  FORCE'S  SELECTION/CLASSIFICATION  SYSTEM  HAS  BEEN  VALIDATED  BY  COMPARING 
AN  INDIVIDUAL'S  SCORES  ON  THEIR  TESTS  TAKEN  PRIOR  TO  ENTERING  THE  SERVICE  (I.E.,  THE  ARMED 
SERVICES  VOCATIONAL  APTITUOE  BATTERY  -  ASVAB)  TO  THEIR  END-OF -TRAINING  GRADE. 

WHAT  WE'RE  ATTEMPTING  TO  00  IN  THIS  PROJECT  IS  TAKE  THAT  RATIONALE  A  STEP  FURTHER  BY  SEEING 
HOW  THEIR  LEVEL  OF  PERFORMANCE  ON  THE  JOB  COMPARES  WITH  THEIR  ENTRY-LEVEL  TEST  SCORES. 


SLIDE  >4  -  RESEARCH  REQUIREMENTS 

THE  AIR  FORCE  ANO  OTHER  SERVICES  ARE  UNOER  A  CONGRESSIONAL  MANOATE  TO  TEST  THE  FEASIBILITY  OF 
LINKING  ENLISTMENT  STANOAROS  TO  JOB  PERFORMANCE.  IN  ADDITION,  THE  AIR  FORCE  COMMUNITY  (MILITARY 
PERSONNEL  CENTER,  AIR  TRAINING  COMMAND,  ANO  AIR  FORCE  CIVILIAN  PERSONNEL  MANAGEMENT  CENTER) 
REQUESTED  THAT  THE  AFHRL  DEVELOP  PERFORMANCE  fCASURES  AS  A  MEANS  OF  EVALUATING  SELECTION, 
CLASSIFICATION,  AND  TRAINING  PROCEDURES,  AS  WELL  AS  SELECTION  PROCEDURES  USED  TO  HIRE  AIR  FORCE 
CIVILIAN  EMPLOYEES. 
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SHOE  »S  -  JOINT-SERVICE  RESEARCH  STRATE6V 


THE  AIR  FORCE,  ALONG  WITH  THE  OTHER  SERVICES,  IS  INVOLVEO  IN  A  COMPLEMENTARY  EFFORT  TO 
DEVELOP  MEASURES  OF  JOB  PERFORMANCE.  CONSEQUENTLY,  THERE  IS  BOTH  OVERLAP  ANO  UNIQUENESS  IN  THE 
SERVICES'  RESEARCH  PROGRAMS.  FOR  EXAMPLE,  EACH  SERVICE  IS  DEVELOPING  WORK-SAMPLE  TESTS.  IN 
AODITION,  THE  ARMY  IS  FOCUSING  ON  JOB  KNOWLEDGE  TESTS,  WHEREAS  THE  NAVY  IS  DEVELOPING  JOB 
SIMULATION  METHODS. 

THE  AIR  FORCE  IS  CONCENTRATING  ON  A  COMBINED  HANDS-ON  ANO  INTERVIEW  APPROACH  KNOWN  AS 
WALK-THROUGH  PERFORMANCE  TESTING  (WTPT).  IN  AOOITION,  A  VARIETY  Of  DIFFERENT  RATING  FORMS  ARE 
BEING  DEVELOPED,  AS  ARE  WORK  EXPERIENCE  MEASURES. 

AS  PART  OF  THIS  JOINT-SERVICE  EFFORT,  WE  ARE  ALSO  PLANNING  TO  SHARE  THESE  NEW  TECHNOLOGIES  TO 
SEE  IF  THE  INSTRUMENTS  HAVE  CROSS-SERVICE  APPLICABILITY.  FOR  EXAMPLE,  THE  WTPT  BEING  DEVELOPED 
FOR  THE  AIR  FORCE  JET  ENGINE  MECHANIC  SPECIALTY  WILL  BE  TRANSFERRED  TO  THE  NAVY  'OR  USE  WITH 
THEIR  JET  ENGINE  MECHANIC  RATING.  IN  AOOITION,  EACH  SERVICE  IS  THE  LEAD  FOR  DIFFERENT 
SPECIALTIES. 


SLIOE  *6  -  AIR  FORCE  LEAD  SPECIALTIES 


THE  AIR  FORCE'S  JOINT-SERVICE  COMMITMENT  COVERS  DEVELOPMENT  ANO  TEST IN6  OF  ttASuREHENT 
HETHOOS  IN  EIGHT  SPECIALTIES: 

1.  426X2  -  JET  ENGINE  MECHANIC  V 

2.  492X1  -  INFORMATION  SYSTEMS  RAO 10  OPERATOR 

3.  272X0  -  AIR  TRAFFIC  CONTROL  OPERATOR 

4.  328X0  -  AVIONIC  C0MRJNICAT10NS  SPECIALIST 

5.  423X5  -  AEROSPACE  GROUNO  EQUIPMENT  SPECIALIST 

6.  732X0  -  PERSONNEL  SPECIALIST 

7.  122X0  -  AIRCREW  LIFE  SUPPORT  SPECIALIST 

8.  324X0  •  PRECISION  KASURING  EQUIPMENT  LABORATORY  SPECIALIST 


SLIOE  #7  -  AIR  FORCE  EFFORT 

THIS  JOINT-SERVICE  COMMITMENT  IS  ONLY  A  PORTION  OF  A  LARGER  PERFORMANCE  MEASUREMENT  EFFORT 
BEING  CONDUCTED  BY  AFHRL. 

CURRENTLY,  CANOIDATE  MEASURES  FOR  AIR  FORCE  SPECIALTY  (AFS)  426X2,  JET  ENGINE  MECHANIC,  ARE 
BEING  FINALI2E0.  THE  FOCUS  ON  A  JOB  PERFORMANCE  MEASUREMENT  SYSTEM  (JPMS)  COMPOSED  OF  WTPT  ANO  { 

RATING  FORMS  HAS  CONTINUED  FOR  SEVEN  MORE  ENLISTED  SPECIALTIES. 

THESE  DEVELOPMENTAL  EFFORTS  WILL  THEN  EXPANO  INTO  32  A001TI0NAL  ENLISTED  SPECIALTIES,  AS  WELL  J 

AS  10  CIVILIAN  ANO  8  OFFICER  SPECIALTIES. 

ONCE  THE  DEVELOPMENT  ANO  VALIDATION  PROCESS  HAS  BEEN  COMPLETED,  THE  MEASUREMENT  AND  • 

VALIDATION  TECHNOLOGIES  WILL  BE  TRANSFERRED  TO  AIR  FORCE  USERS  (ENLISTED  ANO  OFFICER  BY  FY91;  ‘ 

CIVILIAN  BY  FY92). 
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SL 106  »a  -  JOB  PERFORMANCE  MEASUREMENT  SYSTEM 


let  me  explain  in  mope  detail  the  COMPONENTS  Of  the  job  performance  measurement  SYSTEM  (AS 
OEVELOPEO  FOR  THE  JET  ENGINE  MECHANIC). 


SLIDE  »9  -  PERFORMANCE  MEASUREMENT  SYSTEM 


I’LL  TALK  ABOUT  THE  TWEE  MAJOR  COMPONENTS  OF  THE  SYSTEM  —  WTPT,  RATING  FORMS,  AND 
QUESTIONNAIRES  TARGETED  TO  VARIOUS  FACTORS  RELATED  TO  PERFORMANCE  MEASUREMENT  -  AS  WELL  AS 
DESCRIBE  OUR  PLANS  FOR  IMPLEMENTATION  OF  THIS  SYSTEM. 


SLIDE  »10  -  WHAT  IS  A  WALK -THOUGH  PERFORMANCE  TEST? 

WALK-THROUGH  PERFORMANCE  TESTING  CONSISTS  OF  TWO  MAJOR  COMPONENTS:  (A)  HANDS-ON  ITEMS  — 
WHERE  THE  INCUMBENT  IS  REQUIREO  TO  PERFORM  CERTAIN  TASKS,  AND  (B)  INTERVIEW  ITEMS  -  WHERE  THE 
INCUMBENT  IS  REQUIREO  TO  "SHOW  ANO  TELL"  HOW  TO  PERFORM  CERTAIN  TASKS.  IN  AOOITION  TO  HAVING 
SOME  UNIQUE  HANOS-ON  ANO  INTERVIEW  ITEMS.  WE  HAVE  DEVELOPED  OVERLAP  ITEMS  (TASKS  MEASURED  BY  BOTH 
A  HANOS-ON  ITEM  ANO  AN  INTERVIEW  ITEM).  IN  THIS  WAY,  WE  CAN  TEST  THE  COMPARABILITY  OF  THESE  TWO 
APPROACHES. 

BY  USING  BOTH  COMPONENTS.  WE  FEEL  WE  ARE  TAKING  ADVANTAGE  OF  THE  "BEST  OF  BOTH  WORLDS."  WE 
BELIEVE  THE  HANOS-ON  TESTING  PROVIDES  THE  8EST  MEASURE  OF  PERFORMANCE  ON  THE  JOB,  ANO 
INTERVIEWING  PERMITS  ASSESSMENT  OF  THOSE  TASKS  UNABLE  TO  BE  MEASURED  BY  HANOS-ON  TESTING  BECAUSE 
OF  TIME  CONSTRAINTS.  SAFETY  CONSIDERATIONS,  OR  POSSIBLE  EQUIPMENT  DAMAGE.  IN  THIS  HAY.  INTERVIEW 
ITEMS  ALLOW  US  TO  EXPANO  THE  TESTING  OOMAIN. 


SLIDE  »11  -  OVERVIEW  OF  WTPT  DEVELOPMENT 

NOW,  BRIEFLY.  LET  ME  GIVE  YOU  SOME  AOOITIONAL  BACKGROUND  ON  THE  DEVELOPMENT  OF  THE  WTPT 
METHODOLOGY.  I'LL  TOUCH  ON  HOW  TASKS  WERE  SELECTED,  ANO  HOW  ITEMS  WERE  DEVELOPED  ANO  FINALIZED. 


SLIDE  #12  -  TASK  SELECTION 


FIRST,  A  TASK  SELECTION  PLAN  WAS  OEVELOPEO  THAT  WOULD  SELECT  TASKS  REPRESENTATIVE  OF  THE  WORK 
PERFORMED  IN  ANY  SPECIALTY  BY  FIRST-TERM  AIRMEN.  THIS  PLAN  HAS  IMPLEMENTED  IN  THE  JET  EN6INE 
MECHANIC  SPECIALTY,  RESULTING  IN  THE  IDENTIFICATION  OF  THREE  ENGINE  TYPES  (J-57,  J-79,  TF-33)  ANO 
TWO  FUNCTIONAL  AREAS  (SHOP  ANO  FLIGHTLINE).  TASKS  WERE  IDENTIFIED  WITHIN  THESE  CATEGORIES  WHICH 
SUBJECT-IttTTER  EXPERTS  (SMES),  ACTIVE-DUTY  JET  ENGINE  MECHANICS,  REVIEWED  TO  ENSURE  THEY 
REPRESENTED  THE  WORK  PERFORMED  BY  FIRST-TERM  AIRMEN. 

SHOE  >13  -  ITEM  DEVELOPMENT 


ONCE  TASKS  MERE  SELECTEO,  THE  REAL  DEVELOPMENT  WORK  BEGAN.  TASK  ANALYSTS  VISITED  NUMEROUS 
BASES  ANO  INTERVIEWED  SMES.  ON  THE  BASIS  OF  THIS  INFORMATION.  ITEMS  WERE  WRITTEN  FOR  THESE 
TASKS.  THESE  BASE  VISITS  ALSO  ALLONEO  THE  DEVELOPERS  TO  REVIEW  EQUIPMENT  NEEDS,  WORK  SITE 
REQUIREMENTS,  ANO  THE  ADEQUACY  OF  TECHNICAL  ORDERS  ANO  JOB  GUIOES. 

IN  AOOITION  TO  THIS  ON-SITE  WORK,  A  WORKSHOP  HAS  HELO,  WHERE  SMES  WERE  ABLE  TO  ASSIST  IN  THE 
ITEM  DEVELOPMENT  PROCESS  ANO  TO  REVIEW  PROGRESS  TO  OATE. 


SHOE  #14  -  ITEM  FINALIZATION 


AFTER  ITEMS  WERE  DEVELOPED  FOR  ALL  TASKS  SELECTED,  THEY  WERE  PILOT-TESTED  FOR  ACCURACY.  IN 
AOOITION,  INFORMATION  WAS  GATHERED  CONCERNING  EFFICIENT  AND  EFFECTIVE  ADMINISTRATION  PROCEDURES. 

SECOND,  A  FINAL  SET  OF  ITEMS  WERE  SELECTED  FROM  THE  AVAILABLE  POOL  OF  ITEMS.  CRITERIA  USED 
WERE  (A)  RATINGS  OF  ITEM  QUALITY.  (B)  TASK  DIFFICULTY,  (C)  LOGISTICAL  REQUIREMENTS,  AND  (D) 
WHETHER  ITEMS  EXISTED  FOR  BOTH  HANOS-ON  AND  INTERVIEW  TESTING. 

FINALLY,  AN  ITEM  VALIDATION  AND  SCORING  WORKSHOP  WAS  HELD  AT  AFHRL  TO  RECEIVE  FINAL  SME 
APPROVAL  OF  ITEMS  AND  DETERMINE  STEP  CRITICALITY  AND  IMPORTANCE  WITHIN  EACH  ITEM. 


SLIDE  #15  (A  -  D)  -  HANDS-ON  TASK  347 

HERE  IS  AN  EXAMPLE  OF  A  HANOS-ON  ITEM  (NOTE  TO  TRAINER:  POINT  OUT  THE  INDIVIDUAL  COMPONENTS). 


SLIDE  #16  (A  -  D)  -  INTERVIEW  TASK  325 

HERE  IS  AN  EXAMPLE  OF  AN  INTERVIEW  ITEM  (NOTE  TO  TRAINER:  POINT  OUT  THE  INDIVIDUAL 
COMPONENTS). 


SLIDE  #17  -  RATIN6  FORM  DEVELOPMENT 

NOW  LET  ME  PROVIDE  SOME  DETAILS  ASSOCIATED  WITH  RATING  FORM  DEVELOPMENT. 


SLIDE  #18  -  BACKGROUND/APPROACH 

THE  PURPOSE  OF  RATING  FORM  DEVELOPMENT  IS  TO  GENERATE  AND  EVALUATE  ALTERNATIVES  TO  THE  COSTLY 
WTPT. 

FOR  THIS  REASON,  FOUR  DIFFERENT  TYPES  OF  RATING  FORMS  (DIFFERING  BY  LEVEL  OF  SPECIFICITY) 
WERE  DEVELOPED.  THESE  FOUR  RATING  FORMS  ARE  TO  BE  COMPLETED  BY  SUPERVISORS,  INCUMBENTS,  AND 
COWORKERS. 

IT  HAS  ALSO  OECIOEO  THAT  ALL  RATING  FORMS  SHOULD  BE  COMPARABLE  IN  TERMS  OF  NUMBER  OF  RATING 
LEVELS.  THIS  CONSISTENCY  WOULO  ENHANCE  EASE  OF  USE  ACROSS  FORMS. 

THE  PROCEDURE  FOR  THE  DEVELOPMENT  OF  THESE  FORMS  INVOLVED  EXTENSIVE  INTERVIEWS/WORKSHOPS  WITH 
SMES.  IT  WAS  BELIEVEO  SME  INPUT  WAS  CRUCIAL  TO  THE  ACCURATE  DEVELOPMENT  AND  SUBSEQUENT 
ACCEPTANCE  OF  THESE  FORMS  BY  THE  USERS  (I.E.,  JET  ENGINE  MECHANIC  SUPERVISORS,  COWORKERS,  AND 
INCUMBENTS). 
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SLIDE  #19  -  RATING  FORM  DEVELOPMENT 


AS  NOTED,  FOUR  LEVELS  OF  RATING  FORMS  WERE  DEVELOPED: 

1.  TASK:  RATINGS  MAOE  ON  TASKS  REPRESENTATIVE  OF  WORK  PERFORMED  BY  FIRST-TERMERS  IN  THE 
SPECIALTY. 

2.  DIMENSIONAL:  RATINGS  MAOE  ON  AREAS  OF  WORK  (CLUSTERS  OF  TASKS)  IMPORTANT  TO  THE  JOB  OF  A 
FIRST-TERMER. 

3.  GLOBAL:  OVERALL  RATINGS  OF  AN  INDIVIDUAL'S  TECHNICAL  PROFICIENCY  AND  INTERPERSONAL 
PROFICIENCY. 

4.  AIR  FORCE-WIDE:  RATINGS  OF  PERFORMANCE  FACTORS  IMPORTANT  FOR  ALL  AIRMEN. 


SLIDE  »2CA  -  TASK  RATING  FORM 

THE  TASK-LEVEL  RATING  FORM  REPRESENTS  THE  MOST  SPECIFIC,  DETAILED  FORM  DEVELOPED.  TASKS 
REPRESENTATIVE  OF  WORK  PERFORMED  8Y  FIRST-TERMERS  WERE  SELECTED,  INCLUDING  ALL  TASKS  MEASURED  BY 
WTPT. 

BECAUSE  A  RATER  IS  REQUIRED  TO  EVALUATE  AN  INCUMBENT  ON  EVERY  TASK  ON  THIS  RATING  FORM 
(APPROXIMATELY  40  TASKS),  THE  DECISION  WAS  MAOE  TO  LIMIT  THE  REQUIRED  RATINGS  TO  A  5-POINT 
GRAPHIC  RATING  SCALE  (NOTE  TO  TRAINER:  SHOW  SLIDE  206,  AND  READ  ALOUD  THE  FIVE  ADJECTIVAL 
ANCHORS). 

TO  ENSURE  THE  ACCEPTABILITY  ANO  ADEQUACY  OF  THIS  APPROACH,  THE  PROPOSEO  RATING  FORM  WAS 
REVIEWED  BY  SEVEN  SEPARATE  GROUPS  OF  JET  ENGINE  MECHANICS. 


SLIDE  #21 A  -  DIMENSIONAL  RATING  FORM 

THE  NEXT  LEVEL  OF  DETAIL  IS  REPRESENTED  BY  THE  DIMENSIONAL  RATING  FORM.  HERE,  RATERS  ARE 
ASKED  TO  EVALUATE  INCUMBENTS  ON  A  NUMBER  OF  IMPORTANT  AREAS  OF  WORK  PERFORMANCE.  AS  NOTED 
PREVIOUSLY,  BECAUSE  OF  THE  DESIRE  TO  MAINTAIN  CONSISTENCY  ACROSS  RATING  FORMS,  THE  5-POINT  RATING 
SCALE  WAS  USED  ONCE  AGAIN,  WITH  THE  SAME  FIVE  ADJECTIVAL  ANCHORS.  IN  ADDITION,  TO  PROVIDE  THE 
RATER  WITH  MORE  INFORMATION,  BEHAVIORAL  SUMMARY  STATEMENTS  WERE  INCLUDED.  THIS  ADDITIONAL  DETAIL 
HELPS  THE  RATERS  TO  DEVELOP  A  COMMON  "FRAME-OF-REFERENCE"  AS  TO  WHAT  "5"  OR  "3"  ACTUALLY 
REPRESENTS  (NOTE  TO  TRAINER:  SHOW  SLIDE  218). 

BECAUSE  IT  HAS  BELIEVEO  THAT  JET  ENGINE  MECHANIC  SMES  COULD  GIVE  THE  MOST  ACCURATE 
INFORMATION  ABOUT  WHAT  INFORMATION  BELONGS  IN  EACH  OF  THE  FIVE  LEVELS,  A  NUMBER  OF  WORKSHOPS  WERE 
HELD  TO  GATHER  THIS  INFORMATION.  AT  EACH  WORKSHOP,  SMES  "BUILT"  ANO  REVIEWED  THE  FORM.  GIVING 
SUGGESTIONS  FOR  WHAT  AREAS  WERE  IMPORTANT,  WHAT  SPECIFIC  DETAILS  SHOULD  BE  INCLUDED,  ANO  WHAT 
CHANGES  NEEDED  TO  BE  MADE. 

THE  FINAL  RATING  FORM  CONSISTED  OF  THESE  DIMENSIONS: 

1.  COMPLETION  OF  FORMS 

2.  REMOVE/REPLACE  ENGINE  COMPONENTS 

3.  TROUBLESHOOT  (A  FLIGHTLINE  DIMENSION  ONLY) 

4.  SHOP  MAINTENANCE /FLIGHTLINE  MAINTENANCE 

5.  INSPECT  ENGINE 

6.  PREPARATION  FOR  STORAGE  ANO  SHIPMENT  (A  SHOP  DIMENSION  ONLY) 

7.  QUALITY  CONTROL 


v,V',y./u  vV/  v. 
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SLIDE  #22A  -  GLOBAL  RATING  FORM 


THE  DEVELOPMENT  RATIONALE  FOR  THE  GLOBAL  RATING  FORM  WAS  IDENTICAL  TO  THAT  USED  FOR  THE 
OIMENSIONAL  RATING  FORM.  WITHIN  GENERAL  GUIDELINES,  THE  FORM  WAS  CONSTRUCTED  AND  REVISED  BY 
SMES,  AND  PILOT-TESTED  USING  SMES.  AS  NOTED  PREVIOUSLY,  THE  FORM  CONSISTS  OF  TWO  COMPONENTS, 
TECHNICAL  AND  INTERPERSONAL  PROFICIENCY  (NOTE  TO  TRAINER:  SHOW  SLIDES  22B  &  C). 


SLIDE  »23A  -  AIR  FORCE-WIDE  RATING  FORM 


ONCE  AGAIN,  SIMILAR  DEVELOPMENTAL  ACTIVITIES  OCCURRED  WITH  THIS  RATING  FORM.  THE  CONCERN 
WITH  THE  AIR  FORCE-WIDE  RATING  FORM  WAS  TO  IDENTIFY  ENLISTEE  PERFORMANCE  FACTORS  IMPORTANT  ACROSS 
ALL  AIR  FORCE  CAREER  FIELDS.  IN  ORDER  TO  IDENTIFY  THESE  FACTORS,  TWO  HALF-DAY  WORKSHOPS  WERE 
HELD  AT  THE  MILITARY  PERSONNEL  CENTER.  FUNCTIONAL  MANAGERS  FROM  A  WIOE  VARIETY  OF  CAREER  FIELDS 
WERE  ASKED  TO  ATTENO  AND  SERVE  AS  SMES.  THE  FINAL  RATING  FORM  CONSISTED  OF  EIGHT  PERFORMANCE 
FACTORS: 

1.  TECHNICAL  KNOWLEDGE/SKILL 

2.  INITIATIVE/EFFORT 

3.  KNOWLEDGE  OF  AND  ADHERENCE  TO  REGULATIONS/ORDERS 

4.  INTEGRITY 

5.  LEADERSHIP 

6.  MILITARY  APPEARANCE/PHYSICAL  CONDITION 

7.  SELF -DEVELOPMENT 

8.  SELF-CONTROL 

THIS  FORM  HAS  DEVELOPED  UTILIZING  A  FORMAT  SIMILAR  TO  THE  DIMENSIONAL  ANO  GLOBAL  FORMS  (NOTE 
TO  TRAINER:  SHOW  SLIDE  23B). 


SLIDE  #24  -  COVARIABLE  ASSESSMENT 

NOW.  LET  ME  BRIEFLY  MENTION  ADDITIONAL  INFORMATION,  BESIDES  WTPT  AND  RATING  FORMS,  THAT  WILL 
BE  COLLECTED  IN  THIS  EFFORT. 


SLIDE  #25  -  OBJECTIVE  OF  COVARIABLE  ANALYSIS 


THE  INTENT  IN  FOCUSING  ON  COVARIABLE  ANALYSIS  IS  TO  IDENTIFY  FACTORS  RELATED  TO  (I.E.,  THINGS 
THAT  CAN  INFLUENCE)  JOB  PERFORMANCE  OR  ITS  MEASUREMENT. 


SLIDE  *26  -  FACTORS  RELATEO  TO  PERFORMANCE 

FOR  EXAMPLE,  AN  INOIVIOUAL'S  APTITUDE  LEVEL,  AMOUNT  OF  TRAINING  OR  JOB  EXPERIENCE, 
MOTIVATION,  OR  FACTORS  IN  THE  ENVIRONMENT  BEYOND  ONE'S  CONTROL  CAN  AFFECT  HOW  WELL  THE  JOB  IS 
DONE. 


SLIDE  >27  -  FACTORS  RELATED  TO  PERFORMANCE  MEASUREMENT 

IN  AOOITION,  HOW  ACCURATE  THE  MEASUREMENT  DEVICES  ARE  OR  ARE  PERCEIVED  TO  BE  BY  THE  USERS  MAY 
AFFECT  THE  DATA  COLLECTED.  ALSO,  WHETHER  THESE  INSTRUMENTS  ALLOW  DISCRIMINATION  BETWEEN  GOOD  AND 
POOR  PERFORMERS  ANO  WHETHER  THE  INSTRUCTIONS  ARE  CLEAR  AND  UNDERSTANDABLE  CAN  AFFECT  THE  ABILITY 
TO  MEASURE  PERFORMANCE  ACCURATELY. 
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SHOE  »2BA  -  SOURCES  OF  INFORMATION 

THIS  INFORMATION  KILL  8E  COLLECTED  BY  TWO  GENERAL  METHODS.  FIRST,  EXISTING  COMPUTER  FILES. 
WILL  8E  REVIEWED  AND  PERTINENT  INFORMATION  EXTRACTED.  IN  ADDITION,  QUESTIONNAIRES  HAVE  BEEN 
OEVELOPEO  TO  ASSESS  FACTORS  NOT  AVAILABLE  IN  FILE  DATA,  AS  INDICATED  ON  THE  SLIDES  I  WILL  SHOW 
YOU  NEXT. 

(NOTE  TO  TRAINER:  SHOW  THESE  QUESTIONNAIRES  TO  THE  GROUP) 

SLIDES  1288-C  -  TASK  EXPERIENCE 

SLIDES  #28D-E  -  GENERAL  BACKGROUND.  MOTIVATION/COMMITMENT.  SITUATIONAL  CONSTRAINTS 
SLIDES  »2BF-H  -  RATING  FORM  QUESTIONNAIRE 
SLIDES  I28I-K  -  WTPT  QUESTIONNAIRE 
SLIDES  #29  -  DATA  COLLECTION  PUNS 

NOW,  LET  ME  GIVE  YOU  A  BRIEF  GLIMPSE  OF  OUR  PLANS  FOR  DATA  COLLECTION. 


SLIDE  >30  -  DATA  COLLECTION 

FOR  THE  FULL-SCALE  DATA  COLLECTION  EFFORT,  THREE  TO  FOUR  MONTHS  WILL  BE  REQUIRED,  WITH  VISITS 
TO  THREE  OR  FOUR  BASES  PER  ENGINE.  EACH  BASE  VISIT  WILL  LAST  APPROXIMATELY  THREE  WEEKS.  ONE 
HUNORED  TO  ONE  HUNDRED  TWENTY  AIRMEN  PER  ENGINE  (WHICH  fCANS  25-40  PER  BASE)  WILL  BE  TESTED.  THE 
AIM  IS  TO  TEST  AT  LEAST  TWO  AIRMEN  PER  DAY. 


SLIDE  #31  -  TEST  SCHEDULE 

OUR  ANTICIPATED  SCHEDULE  WILL  CLOSELY  RESEMBLE  THAT  DESCRIBED  FOR  THE  PRETEST.  THREE  DAYS 
ARE  ALLOWED  FOR  SCHEDULING  ANO  ACCOMPLISHMENT  OF  THE  BASE  INTRODUCTORY  BRIEFING,  EQUIPMENT  SETUP, 
RATER  TRAINING,  AND  RATING  FORM  ADMINISTRATION.  THE  TEAM  LEADER  IS  RESPONSIBLE  FOR  BOTH  THE 
RATER  TRAINING  AND  THE  BRIEFING.  IN  AOOITION,  ALL  WTPT  ADMINISTRATION  WILL  BE  ACCOMPLISHED  WITH 
ONE  ADMINISTRATOR  PER  INCUMBENT. 


SLIDE  #32  -  OATA  COLLECTION  BASES 

AT  THIS  TIME  WE'RE  PROJECTING  DATA  COLLECTION  AT  THE  BASES  LISTED  ON  THIS  SLIDE. 


Overview  Briefing  Slides 

Slides  29-32  are  AFS  specific.  The  trainer  should  tailor  these  slides  as  necessary  to 
accomodate  the  AFS  under  consideration. 
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HANDS-ON/INTERVIEW  OVERLAP 


r 


Hands- On 


Task  347 


Phase 

Shop 


J-7  9 


and 


,  J-57,  TF-33 
ign  si ine 


Qb.^etlT*  :  To  evaluate  the  incumbent's  ability  to  install 
st  ar  ter  s. 


Estimated 
Time  L'.s 
Tools  and 


:e:  2=  M  Start: 


tlah  : 


Hat  Eta; 


C0347  HT 

2 g  —  #  T  ime  s  Performed:  Last  Performed: 

C0347HN  C03U7HL 

E.q  u  i  t  se  n.t :  Consolidated  Tool  Kit,  0-150  inch-pound 
Torque  Wrench,  1  0-300  inch-pound  Torque  Wrench, 
Lubricant. 


J-79  (Fighter): 

J-57  (Tanker): 

TF-33  ( P  7 )  (Cargo): 

General  Torquing 


Appropriate  T. 0. : 


1F-4Z-10 
1C-135(K)A-2-UJG-6 
1C-1  41  A-2-4JG-5  or 
1C-141 3-10 

2-1-111  or  1-1  A- 3  or 
specific  engine  torquing  I 
J-79:  2J-J7 9-86-7 WP00100 

J-57:  1C-1 35U)A-2-4JG-1 

TF-33:  1C-141B-10 


Background  Information:  There  are  some  common  steps  for  all 
three  engines  but  each  engine  has  some  unique 
steps.  The  ev alua tion  v ill  be  made  on  the  common 
steps  except  when  indicated.  Differences  include: 

1.  J-57  has  two  cannon  plugs. 

J-79  and  TF-33  (?7)  have  one  cannon  plug. 

2.  J-57  and  TF-33  have  one  nut  on  the  V-clamp. 

J-79  has  two  nuts  on  the  7-clamp 

Two-person  task  when  actually  putting  the  starter 
in  place.  This  is  the  only  task  that  the  incumbent 
will  be  required  to  actually  get  the  technical 
order  from  the  shelf. 


Engine  Configuration:  The  starter  adapter  pad  must  be  on  the 
engine.  The  starter  is  off  the  engine. 

Instructions  : 

Administer  in  the  shop. 

The  Incumbent  MUST  use  the  T.  0. 

Compare  the  incumbent's  response  to  the  correct  answer  for  the 
appropriate  engine. 


Phase  I  J-79,  J-5 
Shop  and  Flightl 


ne 


•5 


Hands-On  Task  3 H? 


SAT  TO  TH 2  INCUMBENT 

GET  THE  T.O.  USED  TO  INSTALL  A  STARTER  AND  THE  T.  0.  FOR  GENERAL 
TORO  UING  PROCEDURES,  THEN  INSTALL  THE  STARTER  USING  THE 
APPROPRIATE  PROCEDURES  FROM  BOTH  T.  0.  s.  FOLLOW  GENERAL 
MAINTENANCE  PROCEDURES  AT  ALL  TIMES.  TELL  ME  I?  TOU  PLAN  TO 
DEVIATE  FROM  THE  T.O.  TOU  MAT  NOT  ASK  ANTONE  TO  HELP  TOU  FIND 
THE  CORRECT  T.O. 


Performed  op  Answered  Correctly  Tea  No 

Did  the  incumbent: 

1.  Obtain  the  appropriate  T.O.  for  the 
starter  installation  and  the  torquing 
procedures  within  10  minutes? 

2.  Hang  the  clamp  per  the  specific  T.  0.  ? 

3.  Lubricate  the  spline? 

4.  Ensure  that  the  starter  was  not  left 
in  an  unsupported  position  (hung  by 
the  shaft)  at  any  time? 

5.  Index  (position)  the  starter  per  the 
appropriate  T.  0.  ? 

J-79:  Breach  at  8  o'clock  position 

J-57:  Breach  at  3  o'clock  position 

TF-33:  Drain  plug  at  6  o'clock  position 

6.  Properly  seat  the  V-3and  Clamp?  _  _ 003^7306 

7.  Torque  the  7-3and  Clamp  per  the' 

appropriate  T.  0.  ?  _  _ C03*7H07 

J-79  Airesearch:  110  to  130  inch-pounds 

J-79  Sunstrand:  65  inch  pounds 

J-57:  65  to  70  inch-pounds 

TF-33  :  60  to  70  inch-pounds 

8.  Install  the  locking  device  on  the  7-Band 

Clamp  per  the  appropriate  T.  0.  ?  _  . _ C03$7K08 


_ C03-7  HO  1 

_ C03 -7  HO  2 

_ CO 3  ^7  HO  3 

_ C03  47  30  4 

_ C03*7H05 


9.  Connect  the  applicable  electrical  connector 
(cannon  plug)?  (Must  not  connect  the 
tachometer  generator  plug  on  the  J-57) 


C03«7H09 


Rimnw 
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Phase  I  J-79,  «I-57  ,  TF-33 
Shop  and  Fiightline 


Hands-On  Task  347 


10.  Use  the  correct  tools  and  materials? 


.C03  47  Hi  0 


STOP  TIME 


NOTE:  TORN  PAGE  FOR  RATING  SCALE 
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Phase  I  J-79,  J-57  ,  TF-33 
Shop  and  Flightline 


Hands-On  Task  347  P 


OVERALL  PERFORMANCE 

5  Far  exceeded  the  acceptable  level  of  proficiency 

4  Somewhat  exceeded  the  acceptable  level  of  proficiency 

3  Met  the  acceptable  level  of  proficiency 

2  Somewhat  below  the  acceptable  level  of  proficiency 

1  Far  below  the  acceptable  level  of  proficiency 
C0347HP 


Interview  Task 


325 


Phase  III  J-57 
FI igh tl ine 


Objective; 


To  evaluate  the  Incumbent's  knowledge  concerning 
the  determination  of  high  oil  consumption  on 
J-57  engines. 


Estimated  Tine:  1 OM  Start 


F  t  n  < 


■ae  Hec: 


Time  Limit 


25  M _ iTimes  Performed 


..a  at 


D0325IT 


3  •  m* 


axasd . 


DO 3  25  IN  DO 3  2  5  IL 

Tools  and  Equipment:  None.  T. 0.  1 C-1 3 5( X ) A-2-4MS-3 , 
page  11-3 1 . 


Background  Information:  The  isolation  of  high  oil  consumption 
can  take  from  one  hour  to  several  days. 


Engine  Configuration; 
Instructions ; 


N/A 


Administer  in  the  shop. 

The  incumbent  may  use  the  T.O.  except  when  indicating  the  oil 
flow  path. 


SAT  TO  THE  INCUMBENT 


I  AM  GOING  TO  ASK  TOD  SOME  QUESTIONS  AB  OD  T  J-57  ENGINE  OIL 
CONSUMPTION.  TOO  MAT  USE  T.O.  1  C  - 1  3  5  (  K  5  A- 2  -  4  MS- 3  AS  A  GUIDE 
WHEN  ANSVESING  ALL  THE  QUESTIONS  EXCEPT  THE  FIRST  QUESTION 
WHIC3  DEALS  WITH  THE  OIL  FLOW  PATH. 


Performed  or  Answered  Correctly 
SHOW  THE  INCUMBENT  THE  OIL  FLOW  CHART 


Te  s 


No 


.  Begi 
tell 
the 
Oil 
pum  p 
sump 
Oil 
AN  SV 

a.  Oil 

b.  Oil 

c.  Oil 

d.  Oil 

e.  Oil 
cav  i 

f.  Scav 

g.  Oil 


Incumbents  order  1-9 


h.  Oil 

i.  Oil 


nning  and  ending  at  the  oil  tank, 
me  the  path  that  the  oil  flows  through 
following  components:  Oil  tank, 

Bypass  valve,  Oil  filter,  Scavenge 
s,  Oil  Jets  for  Bearing  cavities  and 
s,  Oil  pressure  relief  valve, 
cooler,  Oil  pump. 

ER: 

Tank 
Pump 

Pressure  Relief  Valve 
Fil ter 

Jets  for  Bearing 
ties  and  sumps 
enge  Pumps 
Bypass  Valve 
Cooler 
Tank 
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Phase  III  J-57 
Fligntiine 


interview  Taste  325 


3. 


Perf ora 

ed 

or 

Answered  Correctly 

Af  te 

r 

chec 

king  the 

oil 

level,  what  area 

woul 

d 

you 

normally 

ch  e 

ck 

next  to  determine 

the 

so 

ur  ce 

of  high 

oil 

CO 

nsumpti  on? 

AN  SW 

ER 

: 

The 

en 

gi  ne 

cowling. 

Why 

is 

the 

engine  cowl 

lag 

;  normally  one 

of  the 

fir 

st  areas 

to 

be 

Inspected  when 

Ye  S 


No 


.DO  3  2  5  10  2  . 


determining  the 
consumption? 

AN  SVEN: 


source  of  high  oil 


Oil  in  the  cowl 

ing 

would  indi 

ca  te  a 

leak. 

— 

_ D0325I03 

4. 

Name  four  areas 

or 

com  po  ne  nt  s 

other 

than 

the  oil  cooler 

that 

you  would 

ch  eck 

for 

. 

oil  leaks. 

(The  Incumbent 

must 

answer  4 

for  credi t ) . 

- 

_ D0325 10  - 

ANSWER: 

Tes 

No 

a. 

Oil  tank 

-- 

b. 

Angle  drive 

— 

c. 

Oil  pump  accessory 

housing 

— 

- 

d. 

Pressure  lines 

— 

— 

e. 

Scavenge  lines 

-- 

f. 

Engine  inlet 

- — 

fi¬ 

Engine  exhaust 

- 

— 

ll. 

Combustion  case 

'  spl 

, it  line 

- 

i. 

Pressurising  and  Dump  (PAD)  v 

al  ve 

—  — 

... 

5.  What  is  the  purpose  of  performing 
a  breather  isolation  check? 
ANSWER: 

To  determine  the  location  of  the 
internal  oil  leak. 


6.  What  could  possibly  be  wrong  if  after 
performing  a  leak  check,  breather 
check,  and  oil  consumption  run  there 
was  no  visible  sign  of  oil  consumption? 
ANSWER: 

Improper  Servicing 


D0325I05 


.D0325I06 


f. 
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?erf oraed  op  Answered  Correctly 


Other  than  checking  the  servicing  level, 
yourself,  or  asking  the  crew  chief,  what 
other  source  is  available  for  deteraining 
when  the  oil  systea  was  last  serviced? 

AN  SW  £3 : 

Read  Aircraft  Forms  such  as  7 S 1 


.003  2  5  10  7 


What  scale  is  read  when  reading  the 
Breather  Gauge? 

ANSWER: 

Inches  in  Mercury 


.DO325I0  3 


What  reading  would  Indicate  a  restriction 
in  the  scavenge  systea? 

ANSWER: 

A  noraal  oil  breather  pressure  reading 
and  a  high  oil  scavenge  pressure  reading. 


.D0325I09 


10.  What  two  basic  pieces  of  inforaation  would 


you  need  to  deteraine  whether  or  not  you 
had  an  excessive  oil  consumption  condition? 


(The  incumbent  must  mention  both  for  credit)  _  _ 

ANSWER:  les  No 


DO*  25  n  o 


The  number  of  flying  hours 

The  nunber  of  quarts  of  oil  serviced 


STOP  TIME 


NOTE:  TORN  PAGE  FOR  RATING  SCALE 
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Phase  III  J-  57  Interview  Task  325  P 

FI igft  1 1 lne 

OVERALL  PERFORMANCE 

5  Far  exceeded  the  acceptable  level  of  proficiency 

U  Somewhat  exceeded  the  acceptable  level  of  proficiency 

3  Met  the  acceptable  level  of  proficiency 

2  Somewhat  below  the  acceptable  level  of  proficiency 

1  Far  below  the  acceptable  level  of  proficiency 
D0325IP 
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RATING  FORM  DEVELOPMENT 


TASK  RATING  FORM 


Example  of  Task  Racing  Form 


5  —  Always  exceeds  acceptable  level  of  proficiency 

H  --  Frequently  exceeds  acceptable  level  of  proficiency 
3  --  Meets  acceptable  level  of  proficiency 

2  —  Occasionally  meets  acceptable  level  of  proficiency 

1  --  Never  meets  acceptable  level  of  proficiency 


Completes  AFTO  Forms  3&9  (Maintenance  Data  Collection 
Record),  AFTO  Forms  350  (Reparable  Item  Processing  Tag), 
and  AFTO  Forms  781  A  (Maintenance  Discrepancy  and  Work 
Document ) . 

Inspects  engine  plumbing. 

Installs  starters. 

Installs  lockvire. 

Inspects  3000-series  trailers  for  serviceability. 

Inspects  area  for  foreign  object  damage  (FOD)  matter. 

Inspects  first  stage  compressors. 

Places  protective  covers  on  engines. 

Services  engine  starters. 

Transports  engines  to  work  sections. 

Installs  tachometer  generators. 

Inspects  engine  or  accessory  splines. 

Installs  J-79  engine  afterburner  secondary  flaps. 

Installs  J-79  engine  constant  speed  drives  (CSD). 

Installs  J-79  engine  forward  top  anti-icing  ducts. 

Installs  J-79  engine  exhaust  gas  temperature 
thermocouple  harness. 

Installs  J-79  engine  ignition  exciter  boxes. 


DIMENSIONAL  RATING  FORM 
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PREPARATION  FOR  STORAGE  AND 
SHIPMENT  (SHOP) 
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Example  of  Dimensional  Racing  Form 
Diaenjion  2:  R  enov  e/ Re  pi  ace  Engine  Components 


This  refers  to  the  entire  procedure  of  reiov  ing/repl  aei  ng  any  engine 
component,  haggle:  Renov  iag/replaci  ng  complex  system  components' 
such  as  fuel  control  or  CSD;  r  em  ov  i  ng/ re  pi  aci  ng  simple  system 
components  such  as  tachometer  generator. 


Circle  the 

L-ey.els  -Number  Behavioral 


Always  exceeds  5 

acceptable  level 
of  proficiency 


Seaoves/repl  aces  complex  system  components 
without  supervision;  is  able  to  complete  all 
remove/replace  tasks  in  less  than  the 
required  time;  demonstrates  an  exceptional 
understanding  and  knowledge  of  tech  orders 
(TOs). 


hr 


■.» 


Frequently  exceeds  H 
acceptable  level 
of  proficiency 


Removes/replaces  complex  system  components 
with  minimal  supervision;  completes  all 
remove/replace  tasks  within  required  time; 
requires  minimal  supervision  in  use  of  TO 
only  on  certain  tasks. 


Meets  acceptable  3 
level  of 
pr  of i ci ency 


Reaov  es/ re  pi  ace  s  complex  system  components 
with  some  supervision;  removes/replaces 
simple  system  components  without  supervision; 
usually  completes  tasks  within  required  time; 
locates  appropriate  TO  and  usually  uses  it 
without  supervision. 


Occasionally  2 

meets  acceptable 
level  of 
prof ici ency 


Removes/replaces  simple  system  components 
with  some  supervision;  requires  substantial 
supervision  when  removing/replacing  complex 
system  components;  usually  takes  longer  than 
required  time  to  complete  tasks;  able  to 
locate  appropriate  TO  but  must  have 
supervision  in  interpretation  and  use. 


Never  meets 
acceptable  level 
of  pr  of i ci ency 


Fails  to  remove/replace  simple  system 
components  even  with  constant  supervision; 
often  unable  to  determine  appropriate  TO 
for  task;  unable  to  interpret  TO  without 
direct  supervision. 
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Global  Racing  Form 


TECHNICAL.  PROFICIENCY 


Circle  the 
Number 


Always  exceeds 
acceptable  level 
of  proficiency 


Frequently  exceeds 
acceptable  level 
of  proficiency 


Meets  acceptable 
level  of 
prof ici ency 


Occa  si o  nally 
meets  acceptable 
1  ev  el  of 
prof ici ency 


Never  meets 
acceptable  level 
of  pr  of  ici  ency 


Completes  all  sections  of  required  tasks 
without  supervision  and  with  few  minor 
errors;  is  able  to  remove  or  replace 
minor/major  components  with  no  supervision 
and  with  few  or  no  errors;  always  brings 
the  correct  materials  to  worksite. 


Completes  all  sections  of  required  tasks 
with  little  or  no  supervision  and  few 
errors;  is  able  to  remove  or  replace 
minor/major  components  with  little  or  no 
supervision  and  with  few  errors;  frequently 
brings  correct  materials  to  worksite. 


Com  pi  etes  all  sections  of  tasks  wi  th  mini  mal 
supervision  and  an  acceptable  number  of 
errors;  is  able  to  remove  or  replace  minor 
components  with  minimal  supervision  and  an 
acceptable  number  of  errors;  usually  brings 
correct  materials  to  worksite. 


Requires  direct  supervision  or  assistance 
on  certain  tasks  in  order  to  avoid  numerous 
errors;  is  able  to  remove  or  replace  minor 
components  with  some  supervision  or  assistance; 
occasionally  brings  correct  materials  to 
worksl  te. 


Is  unable  to  complete  tasks  without  direct 
supervision;  is  unable  to  remove  or  replace 
minor  components  without  direct  supervision 
or  assistance;  and  never  brings  correct 
materials  to  worksite. 
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Global  Rating  Form 


INTERPERSONAL  PROFICIENCY 


Levels 


Ci  rcl  e  the 
Number 


Always  exceeds 
acceptable  level 
of  proficiency 


Frequently  exceeds  4 
acceptable  level 
of  proficiency 


Meets  acceptable 
level  of 
prof  ici  ency 


Oc  ca  si onai ly 
meets  acceptable 
level  of 
pr  of  ici  ency 


Never  meets 
acceptable  level 
of  pr  of  ici  ency 


Always  works  well  with  all  levels  of 
supervision  and  coworkers;  works  effectively 
on  tasks  requiring  teamwork  or  cooperation; 
willing  to  assist  covorkers  in  completing  a 
high  priority  task  without  being  asked  by 
superv isor . 


Frequently  works  well  with  all  levels  of 
supervision  and  coworkers;  works  effectively 
on  tasks  requiring  teamwork  or  cooperation; 
often  willing  to  assist  coworkers  in  completing 
a  high  priority  task  without  being  asked  by 
superv isi or . 


Cooperates  with  most  supervisors  and 
covorkers;  usually  works  effectively  on 
tasks  requiring  teamwork  or  cooperation; 
will  assist  coworkers  in  completing  a  high 
priority  task  only  if  asked  by  supervisor. 


Cooperates  with  only  a  select  group  of 
supervisors  and  coworkers;  rarely  works 
effectively  on  tasks  requiring  teamwork  or 
cooperation;  reluctantly  assists  coworkers 
in  completing  a  high  priority  task  if  asked 
by  supervisor. 


Is  uncooperative  and  ineffective  when 
working  on  a  task  requiring  teamwork  or 
cooperation;  unreceptive  to  guidance  by 
supervisors  or  coworkers;  doesn't  care 
about  functioning  well  as  a  unit  or  crew 


AIR  FORCE-WIDE  RATING  FORM 
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Example  of  Air  Force-Wide  Rating  Form 


Performance  Factor  2:  Initiative/Effort 


Showing  initiative  and  extra  effort  on  j  ob/  Bias!  o  n/a  ssi  gnn  ent . 


Circle  the 

L  evela  Number  Behavioral  Examples 

Always  volunteers  when  opportunities  arise; 
demonstrates  initiative  promptly  and  effect¬ 
ively;  enthusiastically  works  extra  hours  to 
ensure  completion  of  project;  works  to 
completion  when  situation  becomes  difficult. 

Frequently  volunteers  and  demonstrates 
initiative  when  opportunities  arise;  usually 
performs  with  enthusiasm  despite  difficulty; 
willing  to  work  extra  hours  to  complete 
assignment . 

Volunteers  for  some  assignments;  willing  to 
put  in  extra  effort  and  time  on  priority 
tasks;  does  not  give  up  easily  when  faced 
with  obstacles  or  difficulty. 

Occasionally  2  Seldom  volunteers  or  displays  initiative;  may 

meets  acceptable  avoid  difficult  assignments;  has  a  tendency  to 

level  of  stop  working  when  tired  or  bored;  will  work 

performance  extra  hours  only  when  required. 

Never  meets  1  Displays  no  Initiative  and  never  volunteers 

acceptable  level  for  assignments;  reluctant  to  work  extra 

of  performance  hours;  may  become  hostile  when  asked  to  put 

forth  extra  effort;  performs  Ineffectively 
due  to  lack  of  effort;  gives  up  easily  when 
faced  with  a  difficult  task. 


Meets  acceptable  3 
level  of 
perf  orm  ance 


Always  exceeds  5 

acceptable  level 
of  performance 


Frequently  exceeds  4 
acceptable  level 
of  performance 
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COVARIABLE  ASSESSMENT 
JET  ENGINE  MECHANIC 
(AFS  426X2) 


OBJECTIVE  OF  COVARIABLE  ANALYSIS 
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FACTORS  RELATED  TO  PERFORMANCE 
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C(:PTABILITY  /  DISCRIMINABILITY  / 
XtJFtACY 


II 


Example  of  Task  Experience  Questionnaii 


rm4  eacb  Caak  statement  aod  chink  about  Cba  amount  of  relevant 
om-tbe-Job  experience  you're  had  on  Chat  teak,  excluding 
technical  school  training.  Using  the  seals  provided,  vrita  the 
ausber  corresponding  to  your  response  In  the  space  beside  the 


1.  No  or  almost  none 
••  A  small  amount 
1-  A  moderate  amount 
* *  *  9r**t  amount 

*•  A  very  great  amount 


Csuplecee  a/ TU  / orna  349  Utatnceaance  Data  toiisctioo 
Kecord)  AfTO  forma  350  (taparable  Zten  Processing  Tag), 
or  ATTO  forma  7S1A  (Naintenanco  Diacrepaecy  and  Vork 
Doc 


Znapecta  engine  plumbing 


laataila  atartera 


Installs  lockvire 


Inspects  3000Laerlee  trailers  for  serviceability 


Inspects  area  for  foreign  object  damage  (FOD)  matter 


Inspects  first  stags  compressors 


Pisces  protective  covers  on  engines 


Services  engine  starters 


Transports  engines  to  work  sections 


Installs  tachometer  generators 


Inspects  engine  or  accessory  splines 


Installs  J-57  engine  pressure  ratio  probes 


Installs  J-57  engine  anti-icing  valves 


Installs  J-i7  engine  exnaust  gas  temperature  thermocouple 
harness 


Installs  J-57  engine  ignition  exciter  boxes 


Transfers  J-57  engines  from  4 


trailers 


Installs  J-57  starter  control  valves 


Isolates  J -ii  engine- to-s ire rare  tnroctie  rigging  syscea 
malfunctions  when  throttle  is  out  of  alignment 


Isolate#  J-57  engine fuel system  malfunctions  vnen  cne 
angina  fails  to  obtain  combustion 


Dettrminas  sourcs  of  high  oil  consumption  in  J-57  engines 


Isolates  J-57  engine  starter  system  malfunctions  vbeo  ehe 


23.  Installs  J-57  engine  bleed  sir  syscea  governors  _ 


24.  Attaches  s  sling  aod  hoist  to  J-57  engines  on  s  852C 


aircraft 


each  task  iuccmoc  and  CtUak  about  the  amount  o f  relevant 
oo-tha-job  arpariaoce  you'va  bad  on  that  task,  axduditu 
technical  school  training.  Using  tha  seals  pro vtdad,  vrita  tha 
number  corresponding  to  your  raspoosa  la  tha  apaca  beside  tha 
taak. 


23.  Iaspacea  J-37  engine  aircraft  throttle  controls  for 
fraadoa  of  noveaeot 


Positions  4100  stands  for  J-37  satins  removals  or 
lnatsllatlons  _ _ _ 


27.  Adjusts  operating  J-37  engines  using  a  screwdriver _ 


28.  Drains  J-57  fuel  filters _ 


29.  Inspects  J-57  engines  before  operation  _  _ 


30.  Connects  JET  CAL  test  egulpaenc  to  J-57  engines _ 


31.  Services  J-57  engine  oil  ays ten 


32.  Removes  J-57  Constant  Speed  Drive  (CSD)  oil  coolers 


1.  No  or  almost  none 

2.  A  small  amount 

3.  A  moderate  amount 

4 .  A  great  amount 

5.  A  very  great  amount 


CE.NERAL  5ACKCR0UN0 


HAMS  _ _  SSAH  _ - _ - _ 

Last  First  MI 

The  following  questions  pertain  to  your  work  experience,  your  work  unit,  and 
your  feelings  about  your  job.  This  information  will  ba  usad  for  research 
purposes  only.  Please  ehack/fill  in  aach  blank  aa  accurately  aa  poaaibla. 

1.  Present  Skill  Level:  _  3  Level 

______  5  Laval 

7  Level 

2.  Months  in  present  unit:  _ 

3.  Engine  currently  assigned  to:  _  J—  79 

_  J-57 

_  TF-33 

4.  Months  you  have  been  a  Jet  Engine  Mechanic  on  your  present  engine  system 

(i.e.,  J-79,  J-57,  TF-33):  _ 

5.  Area  where  moat  of  your  work  is  done:  _  Shop 

_  Fiightline 

6.  Months  in  shop _ ;  months  on  flightline _ 

7.  Please  list  any  additional  Jet  Engine  Mechanic  experience  below. 

Engine  Type  Amount  of  Experience  (months) 


8.  In  general,  how  is  morale  in  your  unit?  (Check  One)  _  Extremely  Low 

_  Fairly  Low 

_  Average 

_  Fairly  High 

_  Extremely  High 

For  the  following  questions,  use  the  scale  provided  below  to  respond  to  each 
statement. 

1  -  Strongly  Disagree 

2  -  Disagree 

3  ”  Neither  Agree  nor  Disagree 

4  -  Agree 

5  •  Strongly  Agree 

9.  The  technical  manuals  and  other  written  materials  that  I  use  in  my 
job  are  clear  and  understandable. 

10.  _  The  technical  manuals  and  other  written  materials  that  I  use  in  my 

jo3  are  available  when  I  need  them. 


ill 


11. 


The  cool*  and  equipment  Chat  I  uae  in  ay  job  are  available  when  1 
need  ehsa. 


12.  1  aa  able  to  uae  ay  ski  lie  and  talents  ia  ay  job. 

13.  _  I  get  a  sense  of  accoapliahaenc  froa  ay  job. 

14.  _  I  feel  chac  ay  supervisor  ia  concerned  about  ay  well-being. 

15.  _  I  feel  Chat  ay  supervisor  gives  ae  the  support  that  1  need  to  do  ay 

job. 

16.  _  I  feel  that  ay  job  is  interesting. 

17.  _  I  get  a  senae  of  pride  from  being  in  Che  Air  Force. 

18.  _  I  feel  Chat  ay  job  is  iaporcanc  to  Che  overall  niaaion  of  Che  Air 

Force. 

19.  _  1  aa  satisfied  with  ay  job. 

20.  _  1  feel  a  strong  sense  of  responsibility  Co  ay  unit. 

21.  _  I  perfora  ay  duties  to  the  best  of  ay  abilities. 

22.  _  My  technical  school  training  provided  ae  with  the  basic  skills  and 

knowledges  needed  to  do  ay  job. 

23.  _  My  skill  level  upgrade  training  provides  ae  with  the  additional 

skills  and  knowledges  needed  Co  do  ay  job. 

Additional  Coeaents  (If  referring  co  a  specific  question,  please  give  Question 
nuaber) 
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RATING  FORM  QUESTIONNAIRE 


In  che  following  questions  we  are  interested  in  your  beliefs  about  the 
usefulness  of  the  rating  forms  you  just  completed.  Please  respond  to  each 
statement  using  the  scale  provided  below. 


Not  at 

To  a  small 

To  a  moderate 

To  a  great 

To  a  very 

All 

Extent 

Extent 

Extent 

Great  extent 

/ 

/ 

/ 

/ 

/ 

1 

2 

3 

4 

5 

1.  _  How  motivated  were  you  to  complete  the  rating  forms? 

2.  _  Did  you  find  the  performance  rating  process  interesting? 

3.  _  Did  you  care  how  accurate  your  ratings  were? 

4.  _  Did  you  feel  it  was  important  to  make  accurate  ratings? 

5.  _  Did  you  make  an  "extra  effort"  to  carefully  pay  attention  to  all  of 

the  instructions  and  examples  in  order  to  make  accurate  ratings? 

6.  _  Are  you  satisfied  that  you  made  the  most  accurate  ratings  you 

could? 


7.  _  Based  on  your  experience  in  this  project,  how  important  is  it  to 

you  to  make  any  performance  ratings  you  do  as  accurate  as  you  can? 

8.  _  Do  you  believe  that  the  true  purpose  of  the  ratings  was  the  one 

explained  to  you  during  the  rater  orientation? 

9.  _  Do  you  feel  other  persons  involved  really  tried  to  follow  the 

rules  in  completing  their  ratings? 

10. _  Do  you  feel  other  persons  involved  really  cared  about  making 

accurate  ratings? 


11.  _  Do  you  believe  that  the  ratings  collected  will  be  used  for 

research  purposes  only? 

12.  _  Do  you  think  other  persons  involved  gave  higher  ratings  than 

persons  deserved? 


13. 


14. 


Will  your  supervisor  have  access  to  any  information  about  you 
collected  from  the  rating  forms? 


Do  you  feel  other  persons  were  comfortable  giving  low  ratings  to 
themselves  or  others? 
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*1, 
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Do  the  rating  forms  evaluate  your  job  proficiency  fairly? 

15.  _ Task  Rating  Form 

16.  _  Dimension  Rating  Form 

17.  _  Global  Rating  Form 

18.  _  Air  Force-Wide  Rating  Form 

Are  the  rating  forms  easy  to  use  and  understandable  as  a  means  of  determining 
job  proficiency? 

19.  _  Task  Rating  Form 

20.  _  Dimension  Rating  Form 

21.  _  Global  Rating  Form 

22.  _  Air  Force-Wide  Rating  Form 

Would  you  be  able  to  tell  the  difference  between  good  and  poor  performers  by 
looking  at  the  ratings  they  were  given? 

23.  _  Task  Rating  Form 

24.  _  Dimension  Rating  Form 

25.  _  Global  Rating  Form 

26.  _  Air  Force-Wide  Rating  Form 

If  someone  were  to  look  at  the  ratings  on  the  form,  would  they  be  able  to  get 
a  true  picture  of  the  performance  level  of  the  person  being  rated? 

27.  _  Task  Rating  Form 

28.  _  Dimension  Rating  Form 

29.  _  Global  Rating  Form 

30.  _  Air  Force-Wide  Rating  Form 

Overall,  are  the  rating  forms  acceptable  to  you  as  a  way  to  determine  job 
proficiency? 

31.  _  Task  Rating  Form 

32.  _  Dimension  Rating  Form 

33.  _  Global  Rating  Form 

34.  _  Air  Force-Wide  Rating  Form 
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Overall,  did  you  feel  confident  about  the  ratings  you  made? 

35.  _  Task  Rating  Fora 

36.  _  Dimension  Rating  Fora 

37.  _  Global  Rating  Fora 

38.  _  Air  Force-Wide  Rating  Fora 

In  the  following  questions,  we  are  interested  in  your  beliefs  about  the  rating 
forms  in  comparison  to  one  another.  For  each  question,  please  rank-order  the 

four  rating  forms  using  a  “1"  for  the  best,  a  'V  for  next  best,  ana  so  on. 
For  example,  if  you  feel  that  the  Air  Force-wiae  rating  form  provides  the  most 
accurate  ratings  of  a  person's  performance  you  would  place  a  "1“  in  the  space 
beside  "Air  Force-Wiae  hating  Form."  Similarly,  if  you  feel  that  the  Task 
Rating  Form  is  next  to  the  worst  at  provicing  accurate  ratings,  you  woulo 
place  a  "3"  in  the  space  next  to  "Task  Rating  Form." 

Pleas*  be  sure  to  rank  each  type  rating  fora  for  each  question.  Also  use  each 
ranking  number  (1,2, 3, 4)  only  once  for  each  question. 

Are  they  easy  to  us*  and  understandable? 

39.  _  Task  Rating  Fora 

40.  _  Dimension  Rating  Fora 

41.  _  Global  Rating  Fora 

42.  _  Air  Force-Wide  Rating  Fora 

Can  you  tell  the  difference  between  good  and  poor  performers? 

43.  _  Task  Rating  Fora 

44.  _  Dimension  Rating  Form 

45.  _  Global  Rating  Fora 

46.  _  Air  Force-Wide  Rating  Form 

Can  you  get  a  true  picture  of  someone's  performance  level? 

47.  _  Task  Rating  Form 

48.  _  Dimension  Rating  Fora 

49.  _  Global  Rating  Fora 

50.  _  Air  Force-Wide  Rating  Fora 


WTPT  QUESTIONNAIRE 


NAME _ _  SSAN _ - _ _ 

Last  First  MI 

For  the  following  statements  we  are  interested  in  your  beliefs  about  the 
hands-on/interview  testing.  Please  respond  to  each  statement  using  the  scale 
provided  below. 


Not  at 

All 

/ 

To  a  small 

Extent 

/ 

To  a  moderate 
Extent 
/ 

To  a  great 
Extent 

/ 

To  a 

Great 

very 

extent 

_/ 

1 

2 

3 

4 

5 

1.  _  Did  you  feel  it  was  important  to  perform  well  on  the  test? 

2.  _  Are  you  satisfied  that  you  .performed  as  well  as  you  could  on  the 

test?  * 

3.  _  Does  the  test  provide  a  true  picture  of  your  performance  level? 

4.  _  Did  you  care  how  well  you  performed  on  the  test? 

5.  _  Did  you  find  the  test  interesting? 

6.  _  Did  you  make  an  'extra  effort*  to  carefully  pay  attention  to  all 

of  the  instructions  and  examples  in  oraer  to  perform  well.'' 

7.  _  How  motivated  were  you  to  perform  to  the  best  of  your  ability  on 

the  test? 

8.  _  Do  you  believe  that  the  true  purpose  of  the  test  was  the  one 

explained  to  you  by  the  test  administrator? 

9.  _  Do  you  trust  that  the  information  collected  from  you  in  this 

test  will  be  used  for  research  purposes  only? 

10.  _  Will  your  supervisor  have  access  to  any  information  collected 

about  you  from  this  test? 

Is  the  test  acceptable  to  you  as  a  way  to  determine  30b  proficiency? 

11.  _  Hands-on  test 

12.  _  Interview  test 

Did  the  test  evaluate  your  job  proficiency  fairly? 

13.  _  Hands-on  test 


14 


Interview  test 


Will  the  results  of  this  test  be  useful  to  the  Air  Force? 


15.  _ 

Hands-on 

test 

16. 

Interview 

test 

Is  the  test  easy  to  use  and  understandable  as  a  means  of  determining  job 
proficiency? 

17.  _  Hands-on  te3t 

18.  _  Interview  test 

Could  someone  tell  the  difference  between  good  and  poor  performers  by  looking 
at  the  results  of  the  test? 


19.  _  Hands-on  test 

30.  _  Interview  test  ^ 

If  someone  were  to  look  at  the  results  of  the  test,  could  they  get  a  true 
picture  of  the  performance  level  of  the  person  who  took  the  test? 

21.  _  Hands-on  test 

22.  _  Interview  test 

How  well  did  the  instructions  you  received  at  the  beginning  of  each  section 
prepare  you  to  accomplish  that  section? 

23.  Hands-on  test 


24.  _  Interview  test 

What  improvements  would  you  make  in  the  instructions? 


25.  _  How  well  was  the  importance  of  this  Performance  Measurement 

program  to  the  Air  Force  expressed  in  the  orientiation  you 
received? 

26.  _  will  the  information  collected -for  this  performance  measurement 

program  be  used  for  actual  performance  reports  or  other 
administrative  purposes? 


wrjrjru»uwvvv*»j;ruTVTvwwyTvg%nr 


Think  back  on  all  of  the  rating  forma  and  hands-on/i nterv lew  tasks  you  have 
completed  during  this  project.  Please  rank  the  following  on  their  ability  to 
provide  accurate  and  useful  information  about  an  individual's  performance. 

(1  *  Best,  2  *  Next  Best,  3  =  worst). 

27.  Rating  Forms _ _  28. Hands-on  tests _  29.  Interviews 


Additional  Remarks/Comments: 
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PLATTSBURGH 


Base  Introductory  Briefing  Script 


Prior  to  delivering  this  briefing,  the  trainer  should  point  out  to  workshop  attendants  that 
copies  of  the  slides  ana  script  for  the  briefing  are  provided  in  their  workshop  packets.  While 
only  one  person  per  data  collection  team  is  typically  responsible  for  delivering  this  briefing, 
circumstances  may  at  some  time  require  that  another  team  member  conduct  the  briefing.  For  this 
reason,  it  is  imperative  that  all  team  members  be  prepared.  Following  the  briefing,  the  trainer 
should  set  aside  some  blocks  of  time  to  allow  the  test  administrators  to  study  and  then  practice 
the  briefing.  The  trainer  should  monitor  this  practice  at  all  times. 

Slides  #1,  5,  6,  7,  8,  and  9  contain  information  specific  to  AFS  426X2.  The  narrative  for 
these  slides,  as  well  as  the  slides  themselves,  should  be  tailored  by  the  trainer  to  the  AFS 
under  consideration. 


SLIDE  #1  -  JOB  PERFORMANCE  MEASUREMENT  PROJECT 


GOOD  MORNING/AFTERNOON.  MY  NAME  IS  _ . 

THESE  ARE  MY  FELLOW  TEAM  MEMBERS  _ AND  _ . 

WE  REPRESENT  THE  AIR  FORCE  HUMAN  RESOURCES  LABORATORY  AT  BROOKS  AFB,  TX.  WE  ARE  HERE  AS  PART  OF 
THE  AIR  FORCE'S  JOB  PERFORMANCE  MEASUREMENT  PROJECT.  I'D  LIKE  TO  TAKE  A  FEW  MINUTES  TO  GIVE  YOU 
SOME  DETAIL  ON  THE  PROJECT  AND  TO  TELL  YOU  WHAT  WE  WILL  BE  DOING  DURING  OUR  VISIT  AND  HOW  YOU 
WILL  BE  INVOLVED. 


SLIDE  #2  -  OVERVIEW 


FIRST,  I'D  LIKE  TO  DISCUSS  SOME  REASONS  WHY  THE  AIR  FORCE  IS  CONDUCTING  THIS  PROJECT.  THE 
WORK  THE  AIR  FORCE  HAS  DONE  OVER  THE  LAST  YEAR  IN  YOUR  SPECIALTY  INVOLVES  THE  DEVELOPMENT  OF 
VARIOUS  WAYS  TO  MEASURE  AN  INDIVIDUAL'S  ABILITY  TO  PERFORM  HIS/HER  JOB.  I  WILL  THEN  DISCUSS  OUR 
PLANS  TO  COLLECT  JOB  PERFORMANCE  INFORMATION  WHILE  WE  ARE  HERE,  AND  THE  PAYOFFS  OF  THIS  WORK  FOR 
THE  AIR  FORCE. 


SLIDE  »3  -  BACKGROUND 


WHY  IS  THE  AIR  FORCE  OOING  THIS  WORK 7  THERE  ARE  SEVERAL  MAIN  REASONS.  FIRST,  CONGRESS  HAS 
ASKEO  THE  SERVICES  TO  REEXAMINE  THE  WAY  PEOPLE  ARE  SELECTED  AND  ASSIGNED  TO  SPECIALTIES.  SECOND, 
AIR  FORCE  PERSONNEL  AND  TRAINING  MANAGERS  REQUESTED  PERFORMANCE  MEASURES  TO  ENHANCE  THE 
PREDICTION  OF  JOB  SUCCESS.  THE  AIR  FORCE  IS  INTERESTED  IN  ANSWERING  THESE  QUESTIONS,  AS  WELL  AS 
IN  USING  JOB  PERFORMANCE  INFORMATION  TO  KNOW  WHO  TO  TRAIN  AND  WHAT  TYPE  OF  TRAINING  TO  GIVE. 


SLIDE  #4  -  OBJECTIVES 


THEREFORE,  THE  PRIMARY  OBJECTIVES  OF  OUR  EFFORTS  ARE  TO  DEVELOP  A  PERFORMANCE  MEASUREMENT 
TECHNOLOGY  (A)  TO  EVALUATE  TRAINING  PROGRAMS,  AND  (B)  TO  ASSIST  IN  EVALUATING  THE  SETTING  OF 
ENLISTMENT  STANDARDS. 


SLIDE  »S  -  JOB  PERFORMANCE  MEASUREMENT  SYSTEM 

THE  J08  PERFORMANCE  MEASUREMENT  SYSTEM  THAT  HAS  BEEN  DEVELOPED  FOR  YOUR  SPECIALTY  CONSISTS  OF 
THREE  MAIN  COMPONENTS: 

1.  WALK-TnROUGH  PERFORMANCE  TESTING  (WTPT) 

2.  RATING  FORMS 

3.  FACTORS  RELATED  TO  PERFORMANCE  MEASUREMENT  QUESTIONNAIRES 

WALK-THROUGH  PERFORMANCE  TESTING  CONSISTS  OF  TWO  MAJOR  COMPONENTS:  (A)  HANDS-ON  TESTING  AND 
(B)  INTERVIEW  TESTING.  HANDS-ON  TESTING  REQUIRES  INCUMBENTS  TO  PERFORM  TASKS  THEY  ACTUALLY  DO  ON 
THE  JOB.  INTERVIEW  TESTING  REQUIRES  INCUMBENTS  TO  EXPLAIN  THE  STEPS  INVOLVED  IN  PERFORMING  THE 
TASKS  THEY  ACTUALLY  DO  ON  THE  JOB. 

FOUR  DIFFERENT  TYPES  OF  RATING  FORMS  WERE  DEVELOPED.  THESE  RATING  FORMS  WILL  BE  COMPLETED  BY 
SUPERVISORS,  COWORKERS,  AND  INCUMBENTS.  THESE  RATING  FORMS  ARE: 

1.  TASK  RATING  FORM 

2.  DIMENSIONAL  RATING  FORM 

3.  GLOBAL  RATING  FORM 

4.  AIR  FORCE-WIOE  RATING  FORM 

THROUGH  QUESTIONNAIRES  ADMINISTERED  AT  THE  SAME  TIME  AS  THE  RATING  FORMS,  WE  WILL  ALSO  BE 
COLLECTING  INFORMATION  ABOUT  AN  INCUMBENT'S  EXPERIENCE  ON  EQUIPMENT  AND  SPECIFIC  TASKS,  PRIOR 
TRAINING  HE/SHE  HAS  RECEIVEO,  AND  LEVEL  OF  MOTIVATION. 


SLIDE  *6  -  DATA  COLLECTION 

WE  ARE  CURRENTLY  IN  THE  MIDST  OF  FULL-SCALE  DATA  COLLECTION.  FOR  THE  NEXT  3  TO  4  MONTHS,  WE 
WILL  8E  COLLECTING  DATA  FROM  FIRST-TERMERS  IN  THE  426X2  SPECIALTY.  WE  WILL  VISIT  3-4  BASES  PER 
ENGINE  TYPE  (J-57,  J-79,  AND  TF-33),  FOR  A  TOTAL  OF  9-12  BASES.  WE  WILL  USE  THREE  TEAMS  OF  THREE 
TEST  ADMINISTRATORS  EACH  (ONE  TEAM  PER  ENGINE  TYPE).  EACH  TEAM  WILL  SPEND  APPROXIMATELY  3  WEEKS 
AT  EACH  BASE  AND  WILL  EVALUATE  AT  LEAST  TWO  AIRMEN  PER  DAY.  WE  HOPE  TO  COLLECT  DATA  FROM  A  TOTAL 
OF  100-120  AIRMEN  PER  ENGINE,  USING  WALK-THROUGH  PERFORMANCE  TESTING,  RATING  FORMS,  AND 
QUESTIONNAIRES.  THIS  MEANS  WE  WILL  BE  TESTING  25-40  AIRMEN  PER  BASE  WHO  HAVE  BEEN  ON  THE  JOB 
APPROXIMATELY  6  MONTHS  AND  HAVE  UP  TO  48  MONTHS  OF  SERVICE  TIME. 


SLIDE  »7  -  DATA  COLLECTION  TEST  SCHEDULE 

THIS  IS  WHAT  OUR  TESTING  SCHEDULE  LOOKS  LIKE  FOR  YOUR  BASE  AND  OTHERS  INVOLVED  IN  THE  DATA 
COLLECTION  EFFORT.  (NOTE  TO  TRAINER:  TALK  THROUGH  THIS  SCHEDULE,  STEP  BY  STEP.) 


SLIDE  #8  -  TEST  ADMINISTRATOR  QUALIFICATIONS 


THE  TEST  ADMINISTRATORS  WHO  WILL  BE  COLLECTING  DATA  AT  YOUR  BASE  ARE  ALL  FORMER  JET  ENGINE 
MECHANICS,  WITH  A  TOTAL  OF  52  YEARS  OF  MECHANIC  EXPERIENCE  AMONG  THE  THREE  OF  THEM.  THEIR  RECENT 
TRAINING  HAS  INCLUDED  A  REVIEW  OF  THE  TESTING  INSTRUMENTS  DURING  DEVELOPMENT,  PRACTICE  USING  ThE 
INSTRUMENTS,  AND  AN  INTENSIVE  TRAINING  WORKSHOP. 
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SLIDE  #9  -  LOGISTICAL  REQUIREMENTS 

SGT  _ ,  OUR  POINT  OF  CONTACT  HERE  AT  YOUR  BASE,  HAS  ARRANGED  FOR  US  TO  HAVE 

TWO  ENGINES  DEDICATED  TO  US  DURING  OUR  STAY  HERE  AND  HAS  PROVIDED  ROOM  IN  THE  SHOP  FOR  TEST 
ADMINISTRATION.  hE  HAS  ALSO  GRANTED  US  ACCESS  TO  NECESSARY  TECHNICAL  ORDERS,  TOOLS,  AND  OTHER 
EQUIPMENT  AND  HAS  ARRANGEO  FOR  ROOMS  FOR  RATING  FORM  ADMINISTRATION.  (NOTE  TO  TRAINER:  NOW  IS 
THE  TIME  FOR  THE  TEST  ADMINISTRATORS  TO  COMPLIMENT  THE  POC  ON  ThE  WORK  HE  HAS  DONE  FOR  YOU,  IF 
INDEED  A  COMPLIMENT  IS  IN  ORDER.) 


SLIDE  »1Q  -  AIR  FORCE  PAYOFFS 

WHAT  WILL  THIS  WORK  ACCOMPLISH?  IT  WILL: 

1.  IMPROVE  THE  MEASUREMENT  OF  PERFORMANCE 

2.  IMPROVE  THE  AIR  FORCE'S  SYSTEM  OF  SELECTING,  CLASSIFYING,  AND  TRAINING 
INDIVIDUALS 

3.  INCREASE  MISSION  READINESS 


Base  Introductory  Briefing  Slides 

Slides  #1,  5,  6,  7,  8,  and  9  are  AFS-spec if ic.  The  trainer  should  tailor  these  slides  as 
necessary  to  accommodate  the  AFS  under  consideration. 
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TEST  IOO  TO  120  AIRMEN  PER  ENGINE 


TEST  ADMINISTRATOR  QUALIFICATIONS 
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AIR  FORCE  PAYOFFS 


This  section  provides  scripts  and  slides  for  use  in  training  administrators  on  the  two 
components  of  WTPT:  Interview  Testing  and  Hands-On  Testing.  Two  parts.  Interview  Training  and 
the  Behavior  Modeling  Exercise,  address  the  WTPT  Interview  Testing  component;  a  third,  Hands-On 
Testing  Training  addresses  the  other  WTPT  component. 

Interview  Training  Script 

This  part  of  the  training  focuses  on  proper  interviewing  techniques.  It  will  be  followed  by 
a  behavior  modeling  exercise  for  practicing  those  techniques. 


SLIDE  »1  -  INTERVIEW  TESTING 

EARLIER  IN  THE  WORKSHOP,  WE  NOTED  THAT  WTPT  CONSISTS  OF  TWO  COMPONENTS  —  HANDS-ON  TESTING 
AND  INTERVIEW  TESTING.  INTERVIEW  TESTING  MAKES  UP  APPROXIMATELY  50X  OF  THE  TASKS  IN  WTPT. 
THEREFORE,  IT  IS  IMPERATIVE  THAT  WE  SPEND  SOME  TIME  LEARNING  INTERVIEWING  SKILLS.  OVER  THE  NEXT 
SEVERAL  HOURS,  WE  WILL  ACCOMPLISH  THIS  BY:  (A)  DISCUSSING  THE  BASICS  OF  INTERVIEWING,  (B) 
VIEWING  THE  CONDUCT  OF  A  PROPER  INTERVIEW,  AND  (C)  PRACTICING  THE  INTERVIEW  TECHNIQUES  WE  HAVE 
DISCUSSED. 

EVEN  THOUGH  THE  INFORMATION  BEING  COLLECTED  THROUGH  INTERVIEWING  IS  MUCH  THE  SAME  AS  THAT 
GATHERED  BY  MEANS  OF  THE  HANDS-ON  COMPONENT  OF  WTPT,  IT  IS  MUCH  MORE  DIFFICULT  TO  EVALUATE 
BECAUSE  OF  ITS  SUBJECTIVITY.  IN  HANDS-ON  TESTING,  THE  FOCUS  IS  ON  OBSERVATION  AND  RECORDING;  IN 
INTERVIEW  TESTING,  THE  FOCUS  IS  ON  COMMUNICATION  BETWEEN  INDIVIDUALS. 

THE  AMOUNT  OF  INFORMATION  RECEIVED  FROM  THE  INDIVIDUAL  BEING  TESTED  AND  THE  CLARITY  OF  THAT 
INFORMATION  DEPEND  (TO  A  URGE  EXTENT)  ON  HOW  MUCH  THE  INTERVIEWER  CONTRIBUTES  TO  THE  SITUATION. 
THE  FOCUS  OF  OUR  DISCUSSION  WILL  CENTER  ON  THE  TEST  ADMINISTRATOR'S  ROLE  IN  THE  PROCESS. 


SLIDE  #2  -  PRESENTATION/DISCUSSION 

IN  ORDER  TO  EXAMINE  THE  INTERVIEWING  PROCESS,  WE  WILL  FOCUS  ON  THE  COMPONENTS  LISTED  ON  THIS 
SLIDE. 


SLIDE  »3  -  OPENING  THE  TESTING  SESSION 

WHEN  THE  EXAMINEE  FIRST  ARRIVES  AT  THE  TESTING  SITE,  SPECIAL  EFFORT  NEEDS  TO  BE  MADE  TO  PLACE 
THE  INDIVIDUAL  AT  EASE  ANO  ESTABLISH  RAPPORT.  REMEMBER,  THE  EXAMINEES  COME  TO  THE  TESTING 
SITUATION  AT  A  DISADVANTAGE  —  THEY  KNOW  LITTLE  OR  NOTHING  ABOUT  WHAT  WILL  OCCUR,  OR  WHAT  IS 
EXPECTED  OF  THEM. 

THEREFORE,  AT  THIS  TIME,  YOU  SHOULD  TRY  TO  PLACE  THE  INCUMBENTS  AT  EASE  BY  INTRODUCING 
YOURSELF  ANO  TALKING  BRIEFLY  ABOUT  YOURSELF,  OR  SOME  CASUAL  TOPICS,  BEFORE  TURNING  TO  THE  TASK  AT 
HAND.  THEN,  PRIOR  TO  BEGINNING  THE  WTPT,  YOU  WILL  NEED  TO  EXPLAIN  THE  PURPOSE  OF  THE  PROJECT  AND 
WHAT  WILL  BE  REQUIRED  OF  THE  EXAMINEE  OURING  THE  TESTING  PERIOO. 


SLIDE  #4  -  ASKING  QUESTIONS 

1.  USE  the  wtpt  informally,  but  carefully 

WHEN  ADMINISTERING  THE  WTPT  (AND  ESPECIALLY  THE  INTERVIEW  ITEMS),  PRESENT  THE  QUESTIONS 
IN  AN  INFORMAL  MANNER.  IN  OTHER  WORDS,  USE  A  RELAXED  APPROACH  TO  ASKING  QUESTIONS,  AS  A 
MEANS  OF  PUTTING  THE  EXAMINEE  AT  EASE.  HOWEVER,  DON'T  CONFUSE  INFORMALITY  WITH 
CARELESSNESS.  THOROUGH  FAMILIARITY  WITH  WORDING  AND  THE  PROPER  ORDER  OF  STEPS  ALLOWS  A 
RELAXED  ATMOSPHERE  TO  EXIST.  LACK  OF  FAMILIARITY  RESULTS  IN  RIGIDITY. 

2.  ASK  EVERY  QUESTION 

EVEN  WHEN  THE  EXAMINEE  APPEARS  TO  HAVE  ANSWERED  A  PARTICULAR  QUESTION  (OR  STEP)  AT  AN 
EARLIER  POINT,  8E  SURE  TO  ASK  THE  QUESTION.  ASKING  EVERY  QUESTION  WILL  HELP  YOU 
DETERMINE  WHETHER  OR  NOT  THE  INCUMBENT  KNOWS  THE  CORRECT  ANSWER.  DON'T  RELY  ON  PREVIOUS 
COMMENTS.  ALSO,  AS  YOU  BECOME  MORE  FAMILIAR  WITH  THE  EXAMINEE  AS  THE  TEST  PROGRESSES, 
YOU  MAY  HAVE  A  TENDENCY  TO  STEREOTYPE  THE  INDIVIDUAL  ANO  ANTICIPATE  ANSWERS.  BE  AWARE 
THAT  THIS  BEHAVIOR  MAY  LEAD  TO  SKIPPING  QUESTIONS  OR  MARKING  EXPECTED,  RATHER  THAN  ACTUAL 
ANSWERS. 

3.  DO  NOT  SUGGEST  ANSWERS 

RELATED  TO  THE  PREVIOUS  COMMENT  ABOUT  ANTICIPATING  ANSWERS  IS  THE  CONCERN  WITH  SUGGESTING 
ANSWERS.  WHETHER  IT'S  BECAUSE  YOU'VE  BEGUN  TO  STEREOTYPE  THE  INDIVIDUAL  AND  HIS/HER 
RESPONSE  PATTERNS,  OR  HAVE  GROWN  IMPATIENT  WITH  THE  SLOWNESS  OF  HIS/HER  REPLIES,  YOU  MUST 
BE  CAREFUL  NOT  TO  SUGGEST  AN  ANSWER.  WHETHER  THE  ANSWER  YOU  SUGGEST  IS  RIGHT  OR  WRONG, 
THE  EXAMINEE  FREQUENTLY  WILL  PASSIVELY  ACCEPT  YOUR  STATEMENT  AS  TRUTH  (EVEN  IF  IT  ISN'T). 

4.  PROVIDE  TRANSITION  WHEN  NEEDED 

IN  ORDER  TO  PROVIDE  A  SMOOTH  TRANSITION  BETWEEN  ITEMS  OR  STEPS,  YOU  MAY  NEED  TO  GENERATE 
QUESTIONS  OR  COMMENTS.  MAKE  SURE  THESE  STATEMENTS  ARE  NEUTRAL,  NOT  SUGGESTIVE. 

5.  DO  NOT  LEAVE  ANY  STEP  BLANK 

HERE,  JUST  A  REMINDER:  BE  SURE  TO  COMPLETE  ALL  STEPS  ON  ALL  TASKS.  IF  YOU  HAVE 
QUESTIONS  OR  CONCERNS  ABOUT  ANY  STEP,  FIRST  MARK  THAT  STEP,  THEN  ANNOTATE  YOUR  ANSWER. 


SLIDE  »5  -  OBTAINING  ADEQUATE  ANSWERS/PROBING 

PROBABLY  THE  MOST  IMPORTANT  ROLE  OF  THE  ADMINISTRATOR  IS  TO  ENSURE  THAT  THE  ANSWERS  PROVIDED 
BY  THE  EXAMINEE  ARE  CLEAR  AND  TO-THE-POINT.  TO  ACCOMPLISH  THIS,  FREQUENTLY  YOU  MUST  REDIRECT  THE 
EXAMINEE,  PROBE  FOR  ADDITIONAL  INFORMATION,  OR  ASK  FOR  ELABORATION  OR  CLARIFICATION.  SOME 
DIRECTIONS  FOR  OBTAINING  ADEQUATE  ANSWERS  ARE  AS  FOLLOWS: 

1.  REPEATING  THE  QUESTION 

ONE  OF  THE  MOST  USEFUL  PROBES  IS  REPETITION  OF  THE  ORIGINAL  QUESTION.  WHEN  INFORMATION 
IS  MISUNOERSTOOO  OR  NOT  HEARD,  OR  WHEN  THE  EXAMINEE  WANOERS  FROM  THE  TOPIC,  REPETITION  OF 
THE  QUESTION  WILL  REOIRECT  THE  FOCUS  BACK  TO  THE  RELEVANT  SUBJECT  OF  DISCUSSION. 
RESPONDENTS  WILL  OFTEN  INTERPRET  SUCH  "REPETITION  PROBES"  AS  REQUESTS  FOR  ADDITIONAL 
INFORMATION,  AND  WILL  RESPOND  APPROPRIATELY. 


2.  THE  SILENT  PROBE 

A  SIMPLE,  NEUTRAL  HAY  OF  STIMULATING  FURTHER  DISCUSSION  IS  BY  USING  THE  EXPECTANT  PAUSE. 
WHEN  ACCOMPANIED  BY  AN  EXPECTANT  LOOK  OR  NOD  OF  THE  HEAD,  SILENT  PROBES  ENCOURAGE 
COMMUNICATION,  INEXPERIENCED  INTERVIEWERS  OFTEN  FEEL  UNCOMFORTABLE  WITH  THIS  TECHNIQUE  ' 
8ECAUSE  MOST  OF  ITS  SUCCESS  LIES  IN  THE  ATTITUDES  AND  BEHAVIOR  OF  THE  INTERVIEWER.  A 
CONFIDENT,  NEUTRAL,  YET  EXPECTANT  LOOK  FOSTERS  COMMUNICATION  WITH  THE  EXAMINEE. 

3.  ELABORATION/CLARIFICATION 

ASKING  FOR  ADDITIONAL  INFORMATION  WHEN  AN  INSUFFICIENT  OR  AMBIGUOUS  RESPONSE  HAS  BEEN 
GIVEN  ALLOWS  YOU  TO  DETERMINE  THE  CORRECTNESS  OR  INCORRECTNESS  OF  THE  RESPONSE  WITHOUT 
GUESSING.  HOWEVER,  YOU  MUST  BEWARE  NOT  TO  PROVIDE  LEADING  OR  SUGGESTIVE  INFORMATION  IN 
YOUR  PROBES,  OR  ALLOW  THE  TONE  OF  YOUR  VOICE  TO  INFLUENCE  AN  ANSWER.  SOME  EXAMPLES  OF 
CORRECT  REQUESTS  FOR  ADDITIONAL  INFORMATION  ARE  PROVIDED  IN  THE  NEXT  SLIDE. 


SLIDE  #6  -  NEUTRAL  QUESTIONS  OR  PROBES 

APPROPRIATE  PROBES  TO  USE  DURING  INTERVIEWING  ARE  QUESTIONS  SUCH  AS  THOSE  NOTED  ON  THIS  SLIDE. 


SLIDE  #7  -  RECORDING  ANSWERS/RESPONSES 


THE  BEST  TIME  TO  RECORD  ANSWERS  IS  IMMEDIATELY  AFTER  THE  RESPONSE  HAS  BEEN  GIVEN.  IN 
ADOITION  TO  RECORDING  ANSWERS,  THE  INTERVIEWER  SHOULD  ANNOTATE  EXPLANATORY  COMMENTS  WHEREVER 
NECESSARY,  ESPECIALLY  IF  THE  INFORMATION  PROVIDES  CLARIFICATION.  STILL,  THE  INTERVIEWER  MUST  BE 
ABLE  TO  WRITE  UNDERSTANDABLE  NOTES  WHILE  ENGAGED  IN  CONVERSATION.  FOR  RECORD-KEEPING  PURPOSES, 
RECORD  ANY  ADDITIONAL  COMMENTS  ON  THE  TEST  ITSELF. 


SLIDE  #8  -  CLOSING  THE  TESTING  SESSION 

ONCE  ALL  THE  TASKS  HAVE  BEEN  PERFORMED  BY  THE  INCUMBENT  AND  THE  WTPT  QUESTIONNAIRE  COMPLETED, 
THE  TESTING  SESSION  CAN  BE  CONCLUDED.  THE  PURPOSE  OF  THE  WORK  SHOULD  BE  REITERATED,  AND  THE 
EXAMINEE  THANKED  FOR  HIS/HER  COOPERATION.  EACH  INCUMBENT  SHOULD  ALSO  BE  INSTRUCTED  TO  AVOID 
01 SCUSSING  THE  TESTING  PROCESS  WITH  OTHER  FIRST-TERM  AIRMEN. 


SLIDE  #9  -  INTERVIEWING  DO'S 

HERE  ARE  SOME  SUGGESTED  GUIDELINES  FOR  THINGS  TO  DO  IN  THE  INTERVIEWING  SITUATION. 

1.  SHOW  POLITENESS  ANO  COURTESY  THROUGHOUT  THE  INTERVIEW. 

2.  BE  IMPERSONAL.  DO  NOT  BE  CRITICAL  OR  SUGGEST  CHANGES  OR  IMPROVEMENTS  IN  PROCEDURE i ,  - 

THE  ORGANIZATION. 

3.  ALLOW  THE  WORKER  SUFFICIENT  TIME  TO  ANSWER  EACH  QUESTION. 

4.  CONOUCT  THE  INTERVIEW  IN  PLAIN,  EASILY  UNDERSTOOD  LANGUAGE. 


5.  CONTROL  THE  INTERVIEW  WITH  RESPECT  TO  THE  ECONOMIC  USE  OF  TIME  ANO  Al-c-;S 
MATTER. 


6.  CONDUCT  THE  INTERVIEW  WITH  PATIENCE  ANO  CONSIDERATION  FOR  NERVOUSNESS  ON  THE  PART  OF  THE 
WORKER. 

7.  ENCOURAGE  THE  INCUMBENT  TO  SPEAK  FREELY. 

8.  JOT  DOWN  NOTES  WHENEVER  NECESSARY  TO  CLARIFY  A  POINT  OR  TO  SUPPLY  BACKGROUND  INFORMATION 
WHERE  USEFUL. 

SLIDE  »10  -  INTERVIEWING  DON'TS 

THIS  SLIDE  PROVIOES  SOME  SUGGESTED  GUIDELINES  FOR  ACTIONS  TO  AVOIO  IN  THE  INTERVIEW  SITUATION. 

1.  DO  NOT  SHOW  PARTIALITY  TO  OPINIONS. 

2.  DO  NOT  “TALK  DOWN*  TO  THE  PERSON  YOU  ARE  INTERVIEWING. 

3.  DO  NOT  BE  INFLUENCED  BY  YOUR  BIASES. 

4.  DO  NOT  TAKE  ISSUE  WITH  THE  RESPONDENT'S  COMMENTS. 

5.  DO  NOT  USE  LEAOING  QUESTIONS. 

6.  DO  NOT  COMPARE  EXAMINEES. 
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Interview  Training  Slides 

Slides  #9  and  10  should  be  Included  as  handouts  In  test  administrator  workshop  packets. 
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Behavior  Modeling  Exercise 

A  vital  part  of  test  administrator  interview  training  is  the  modeling  exercise  which  follows 
the  trainer's  lecture  on  interview  techniques.  This  exercise  demonstrates  the  proper  conduct  of 
an  interview  item  via  a  videotape  of  two  actors  who  role-play  an  interview  test  item  and  utilize 
appropriate  communication  and  probing  techniques  throughout  the  interview.  The  actors  also 
demonstrate  appropriate  methods  of  opening  and  closing  a  day  of  testing.  This  exercise  provides 
an  opportunity  for  test  administrators  to  observe  the  modeling  of  appropriate  interview  behavior. 

Interview  item  #134  (from  the  426X2  WTPT)  was  chosen  for  the  exercise  because  it  is  lengthy 
enough  to  allow  the  demonstration  of  a  variety  of  interview  techniques.  The  videotape  can  be 
used  with  other  specialties  since  the  concept  of  good  interviewing  techniques  is  not  AFS-specif ic. 

The  trainer  should  introduce  the  videotape  with  instructions  to  pay  close  attention  to  the 
good  communications  and  probing  skills  demonstrated  by  the  interviewer.  The  videotape  should 
then  be  played  in  its  entirety.  Next,  the  trainer  should  facilitate  discussion  by  replaying  the 
segments  of  the  tape  that  illustrate  the  various  techniques.  This  segment  of  the  workshop 
concludes  with  an  opportunity  for  the  test  administrators  to  role-play  interview  items  in  pairs 
and  thereby  practice  utilizing  proper  interview  procedures.  The  trainer  should  monitor  the 
role-play,  listen  for  the  use  of  appropriate  techniques,  and  provide  corrections  as  necessary. 
After  everyone  has  played  both  roles  (interviewer  and  interviewee),  the  trainer  should  discuss 
any  issues  brought  up  by  the  administrators,  or  observations  noted. 

This  section  contains  the  script  used  in  developing  the  videotape  employed  in  the  modeling 
exercise,  as  well  as  guidelines  for  the  trainer  to  follow  while  conducting  the  discussion. 


Modeling  Exercise  Script 

Step  #  Oialogue 

1.  Interviewer:  What  is  the  purpose  of  the  AFTO  Form  350,  Reparable  Item  Processing 

Tag? 

Respondent:  Controls  the  repair  process  of  a  part. 

2.  Interviewer:  Under  what  conditions  would  a  supply  document  number  entry  be  made  in 

81ock  13  of  AFTO  Form  350? 

Respondent:  When  you  order  a  like  item  from  supply. 

3.  Interviewer:  When  working  in  the  shop,  in  which  block  of  AFTO  Form  350  do  you  note 

that  you  have  repaired  an  item  by  replacing  a  part? 

Respondent:  Block  15. 

4.  Interviewer:  What  is  the  purpose  of  the  Job  Control  Number  on  AFTO  Form  349? 

Respondent:  Identifies  and  controls  maintenance  jobs. 

5.  Interviewer:  What  do  the  first  three  numbers  of  the  job  control  number  on  the  AFTO 

Form  349  represent? 

Respondent:  The  date. 

Interviewer:  CAN  YOU  BE  MORE  SPECIFIC? 

Respondent:  The  calendar  date. 
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Interviewer:  What  does  the  work  center  entry  on  the  AFTO  Form  349  tell  us? 

Respondent:  It  tells  you  the  work  site. 

Interviewer:  WHAT  DO  YOU  MEAN  BY  WORKSITE? 

Respondent:  Where  you  are  assigned  to  work. 

Interviewer:  What  is  entered  in  Block  9  of  AFTO  Form  349  if  the  form  is  used  as  a 

dispatch  form? 

Respondent:  Spot. 

Interviewer:  SPOT? 

Respondent:  Yea,  like  Yankee  2. 

Interviewer:  WHAT  00  YOU  MEAN  BY  YANKEE  2? 

Respondent:  This  is  where  the  aircraft  is  parked.  It's  where  you  go  to  do  your 

work. 

Interviewer:  Under  what  condition  is  the  National  Item  Identification  Number 

(NIIN)  entered  in  Block  20  of  AFTO  Form  349? 

Respondent:  I  don't  understand  the  question. 

Interviewer:  LET  ME  READ  IT  TO  YOU  AGAIN  (repeats  question). 

Respondent:  I  don't  know.  I've  never  used  that  block. 

Interviewer:  What  entry  on  the  AFTO  Form  349  identifies  the  part  on  which  the  work 

is  being  performed? 

Respondent:  Your  work  unit  code,  column  C. 

Interviewer:  Except  for  the  FCF  (Functional  Check  Flight),  who  completes  the  top 

line  entry  on  AFTO  Form  781A? 

Respondent:  The  crew  chief. 

Interviewer:  What  is  the  purpose  of  AFTO  Form  781A? 

Respondent:  It's  a  historical  record  of  all  maintenance  performed  on  the  aircraft. 

Interviewer:  What  color  pencil  is  normally  used  to  fill  in  the  symbol  block  on 

Form  781A  when  there  is  a  discrepancy? 

Respondent:  Red. 

Interviewer:  Besides  signing  in  the  “Inspected  By"  block,  what  else  must  the 

Inspector  complete  to  Indicate  that  a  red  X  condition  has  been 

cleared  on  AFTO  Form  781A? 

Respondent:  Last  name  Initial  over  the  symbol. 

Interviewer:  WHAT  KINO  OF  MARKING  DEVICE  WOULD  YOU  USE? 

Respondent:  A  pencil. 

Interviewer:  How  could  you  tell  by  reading  AFTO  Form  781A  that  a  part  had  been 

ordered? 

Respondent:  What  block  are  you  talking  about? 

Interviewer:  I  CAN'T  TELL  YOU  WHAT  BLOCK  I'M  TALKING  ABOUT.  TAKE  ANOTHER  LOOK  AT 

THE  781A  AND  I'LL  REPEAT  THE  QUESTION  (repeats  question). 

Respondent:  There  would  be  an  order  number  in  the  discrepancy  block? 

Interviewer:  IS  THAT  YOUR  ANSWER? 

Respondent:  Yes. 


15.  Interviewer: 

Respondent: 

Interviewer: 

Respondent: 

Interviewer: 

Respondent: 

Interviewer: 

Respondent: 


What  do  the  following  symbols  mean  when  entered  in  red  in  the  symbol 
block  on  AfTO  Form  781A?  A  red  X? 

Major  problem. 

CAN  YOU  EXPLAIN  TO  ME  WHAT  YOU  MEAN  BY  A  MAJOR  PROBLEM? 

Unsafe  condition. 

A  red  diagonal? 

Unknown  condition. 

A  red  dash? 

Minor  condition. 


NOW  WE  WILL  REPLAY  THE  TAPE  AND  IDENTIFY  THE  TECHNIQUES  USED  BY  THE  INTERVIEWER. 


(NOTE  TO  TRAINER:  SHOW  OPENING  OF  INTERVIEW.) 

NOTICE  THAT  THE  INTERVIEWER  GREETS  THE  RESPONDENT  BY  NAME  AND  THEN  ASKS  THE  RESPONDENT  A 
LITTLE  ABOUT  HIMSELF.  LIGHT  CONVERSATION  SUCH  AS  THIS  IS  USUALLY  QUITE  EFFECTIVE  IN  PUTTING  A 
RESPONDENT  AT  EASE.  THEN  NOTICE  THAT  THE  INTERVIEWER  DEMONSTRATES  HOW  TO  OPEN  A  DAY  OF  TESTING 
FOR  A  WTPT  INCUMBENT. 


(NOTE  TO  TRAINER:  SHOW  STEP  #5.) 

THIS  IS  THE  FIRST  TIME  THE  INTERVIEWER  HAS  TO  USE  A  PROBE.  "CAN  YOU  BE  MORE  SPECIFIC?"  IS  A 
MEANS  OF  OBTAINING  CLARIFICATION  ON  AN  ANSWER. 

(NOTE  TO  TRAINER:  SHOW  STEP  #6.) 

"WHAT  DO  YOU  MEAN  BY  ...  ?“  IS  A  COMMON  ELABORATION  PROBE.  THIS  INTERVIEWER  IS  REQUESTING 
MORE  INFORMATION  IN  A  NEUTRAL,  NON-LEADING  MANNER. 

(NOTE  TO  TRAINER:  SHOW  STEP  #7.) 

THE  INTERVIEWER  USES  A  VERY  EFFECTIVE  TECHNIQUE  WHEN  HE  REPEATS  THE  RESPONDENT’S  ANSWER  - 
"SPOT"  —  AND  THEN  PAUSES  TO  ALLOW  THE  RESPONDENT  TO  ELABORATE  OR  CLARIFY  HIS  ANSWER.  NOTICE 
THAT  THE  INTERVIEWER  DOES  IT,  NOT  IN  A  QUESTIONING  WAY  BUT,  IN  A  VERY  EVEN  TONE  OF  VOICE.  BE 
CAREFUL  TO  ENSURE  THAT  YOUR  FACIAL  EXPRESSIONS  AND  TONE  OF  VOICE  REMAIN  PLEASANT,  BUT  NEUTRAL. 
FOR  EXAMPLE,  YOU  WOULD  AVOID  RAISEO  EYEBROWS  OR  A  QUESTIONING  TONE  WHICH  WOULD  TELL  THE 
RESPONDENT  THERE  IS  SOMETHING  WRONG  WITH  HIS/HER  ANSWER.  THE  INTERVIEWER  HAS  TO  REQUEST  FURTHER 
ELABORATION  ON  THIS  STEP  WITH  “WHAT  00  YOU  MEAN  BY  YANKEE  2?" 

(NOTE  TO  TRAINER:  SHOW  STEP  #8.) 

HERE.  THE  INTERVIEWER  REPEATS  THE  ORIGINAL  QUESTION  BUT  OOES  NOT  TRY  TO  EXPLAIN  THE  QUESTION 
IN  ANY  WAY.  THE  TEST  QUESTIONS  ARE  STANDARDIZED,  AND  YOU  SHOULD  NEVER  ATTEMPT  TO  REPHRASE  THE 
QUESTIONS  OR  GIVE  ADDITIONAL  INFORMATION,  EVEN  IF  THE  RESPONDENT  ASKS  YOU  TO.  DOING  SO  WOULD  BE 
UNFAIR  FOR  THOSE  RESPONDENTS  WHO  00  NOT  RECEIVE  THE  SAME  TREATMENT. 

REPEATING  THE  QUESTION  IS  ALSO  VERY  EFFECTIVE  IF  YOU  FIND  THE  RESPONDENT  WANDERING  FROM  THE 
SUBJECT  OR  GIVING  AN  IRRELEVANT  ANSWER.  SUCH  BEHAVIOR  COULD  INDICATE  THAT  THE  RESPONDENT 
MISUNDERSTOOD  THE  QUESTION. 

(NOTE  TO  TRAINER:  SHOW  STEP  #13.) 

HERE  THE  INTERVIEWER  ASKS  A  VERY  SPECIFIC  QUESTION  ("WHAT  KIND  OF  MARKING  DEVICE  WOULD  YOU 
USE?")  TO  ELICIT  ADDITIONAL  INFORMATION.  THIS  MAY  APPEAR  TO  BE  A  LEADING  QUESTION,  BUT  THOROUGH 
INVESTIGATION  OF  THIS  PARTICULAR  STEP  BY  EXPERTS  IN  THE  JET  ENGINE  MECHANIC  SPECIALTY  INDICATES 
THAT  A  RESPONDENT  ASSUMES  THE  INTERVIEWER  KNOWS  THAT  THE  RESPONDENT  IS  REFERRING  TO  A  BLACK 
PENCIL.  THIS  STEP  REQUIRES,  HOWEVER,  THAT  THE  RESPONDENT  VERBALIZE  THE  MARKING  DEVICE.  IN 
SITUATIONS  SUCH  AS  THIS,  IT  IS  NECESSARY  FOR  TEST  ADMINISTRATORS  TO  REACH  A  CONSENSUS  IN  ADVANCE 
ON  THE  TYPE  OF  PROBE  TO  USE  TO  AVOID  BLATANTLY  LEADING  THE  RESPONDENT. 


(NOTE  TO  TRAINER:  SHOW  STEP  #14.) 
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notice  that  the  interviewer  does  a  good  job  of  maintaining  control  of  the  interview  and  not 

SUCCUMBING  TO  THE  RESPONDENT'S  MANIPULATION  OF  THE  QUESTION.  AFTER  THE  INTERVIEWER  REPEATS  THE 
QUESTION,  THE  RESPONDENT  GIVES  AN  ANSWER  WITHOUT  CONVICTION.  HE  IS  WAITING  FOR  CONFIRMATION  FROM 
THE  INTERVIEWER  THAT  THE  ANSWER  IS  CORRECT.  THE  INTERVIEWER  HANDLES  THIS  VERY  WELL  BY  ASKING, 
"IS  THAT  YOUR  ANSWER?"  THIS  IS  PROBABLY  ONE  OF  THE  MOST  FREQUENT  PROBES  YOU  AS  INTERVIEWERS  WILL 
HAVE  TO  USE.  MANY  FIRST-TERMERS  MAY  FEEL  INSECURE  IN  THEIR  ANSWERS,  AS  THIS  RESPONDENT  DIO. 
OTHERS  MAY  PROVIDE  SEVERAL  ANSWERS  TO  A  QUESTION,  HOPING  THE  INTERVIEWER  WILL  ACCEPT  ONE  OF  THEM 
AS  THE  CORRECT  ANSWER.  YOU  MUST  FORCE  THE  RESPONDENT  TO  MAKE  A  CHOICE. 

(NOTE  TO  TRAINER:  SHOW  STEP  #15.) 

FINALLY,  THE  INTERVIEWER  HAS  TO  ASK  FOR  ADDITIONAL  INFORMATION  TO  CLARIFY  WHAT  THE  RESPONDENT 
MEANS  BY  "MAJOR  PROBLEM." 

(NOTE  TO  TRAINER:  SHOW  CLOSING  OF  INTERVIEW.) 

THE  INTERVIEWER  CLOSES  NOT  ONLY  AN  INTERVIEW,  BUT  ALSO  A  DAY  OF  TESTING.  HE  REITERATES 
IMPORTANT  POINTS,  THANKS  THE  INCUMBENT,  AND  SAYS  FAREWELL.  NOTICE  HE  GIVES  NO  INDICATION  OF  HOW 
THE  INCUMBENT  PERFORMED. 


In  this  third  segment  of  WTPT  training,  test  administrator  teams  are  shown  correct  procedures 
for  performing  all  WTPT  hands-on  tasks.  The  aim  in  these  exercises  is  to  calibrate  the 
observation  and  rating  processes  of  the  administrators.  The  time  required  for  this  training 
component  may  vary  based  on  the  experience  of  the  administrators.  It  is  imperative  that 
administrators  are  rating  WTPT  in  the  same  manner. 

At  this  point  in  the  workshop,  the  large  group  will  separate  into  their  respective  evaluation 
teams.  Separate  rooms  and  videocassette  players  need  to  be  reserved  beforehand  for  this 
exercise.  Approximately  4  hours  will  be  required  for  this  task. 

Videotapes  are  then  shown  for  WTPT  hands-on  tasks,  and  administrators  are  required  to  rate  an 
individual's  performance  on  these  tasks.  A  task  is  shown  being  performed  either  correctly  or 
incorrectly.  After  each  task  is  performed  and  evaluated  in  WTPT  booklets,  the  group  should  reach 
consensus  on  their  ratings  of  each  step  of  the  task,  as  well  as  on  their  overall  rating  of  the 
task.  On  those  tasks  that  were  shown  being  performed  incorrectly,  the  correct  procedures  are 
shown  following  the  group  discussion. 


SLIDE  »1  -  HANDS-ON  TESTING 

WE’VE  SPENT  THE  LAST  SEVERAL  HOURS  FOCUSING  ON  THE  INTERVIEW  TESTING  COMPONENT  OF  WTPT.  WHAT 
I'D  LIKE  TO  DO  NOW  IS  CONCENTRATE  ON  HANDS-ON  TESTING.  THE  PURPOSE  WILL  BE  TO  FAMILIARIZE  EACH 
OF  YOU  WITH  THE  HANOS-ON  TASKS  ON  WHICH  YOU  WILL  BE  EVALUATING  INCUMBENTS.  IN  ADDITION,  BEFORE 
YOU  GO  OUT  ON  THE  PRETEST  AND  FULL-SCALE  DATA  COLLECTION  EFFORT,  WE  NEED  TO  FEEL  COMFORTABLE  THAT 
ALL  OF  YOU  ARE  OBSERVING  AND  EVALUATING  INCUMBENTS  IN  THE  SAME  WAY. 


SLIDE  »2  -  APPROACH 

OUR  APPROACH  T00A\  WILL  CONSIST  OF  THREE  MAIN  ACTIVITIES  THAT  CENTER  ON  THESE  HANDS-ON 
TASKS.  FIRST,  WE  WILL  DIVIDE  INTO  THE  THREE  ENGINE  TEAMS  AND  THEN  VIEW  VIDEOTAPES  OF  JET  ENGINE 
MECHANICS  PERFORMING  EACH  TASK. 

DURING  THE  PERFORMANCE  OF  EACH  TASK,  YOU  SHOULD  FOLLOW  ALONG  IN  YOUR  WTPT  TEST  BOOKLET  AND 
PLACE  CHECKMARKS  JUST  AS  IF  YOU  WERE  EVALUATING.  SOMEONE  DURING  DATA  COLLECTION.  AT  THE 
COMPLETION  OF  EACH  TASK,  THE  TAPE  WILL  BE  STOPPED  AND  YOU  WILL  NEED  TO  GIVE  AN  OVERALL  RATING  FOR 
THAT  TASK. 

AFTER  ALL  OF  YOU  HAVE  COMPLETED  YOUR  EVALUATIONS  ON  THAT  TASK,  WE  WILL  DISCUSS  EACH  STEP  (AS 
WELL  AS  THE  OVERALL  RATING),  FOCUSING  ON  HOW  COMPARABLE  YOUR  RATINGS  ARE  FOR  THAT  TASK.  THE  AIM 
HERE  WILL  BE  TO  REACH  CONSENSUS  8EF0RE  MOVING  TO  A  NEW  TASK.  ALL  TASKS  WILL  BE  OBSERVED  AND 
DISCUSSED  IN  THIS  MANNER. 


Hands-On  Testing  Training  Slides 
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HANDS-ON  TESTING 
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Rater  Training  Session 

Once  they  arrive  at  a  base  for  data  collection,  test  administrators  may  be  responsible  for 
effectively  conducting  a  rater  training  session  for  all  WTPT  incumbents,  as  well  as  the 
incumbents'  supervisors  and  peers,  to  enable  these  individuals  complete  their  rating  forms. 

There  are  three  parts  to  the  rater  training  session.  The  rater  training  briefing  is  the 
first  component.  The  briefing  explains  the  purpose  of  the  visit  and  stresses  the  importance  of 
the  raters'  participation  in  the  project.  This  briefing  is  almost  identical  to  the  8ase 
Introductory  Briefing.  Since  the  rater  training  session  precedes  administration  of  the  WTPT,  the 
briefing  provides  test  administrators  an  opportunity  to  establish  rapport  with  the  incumbents  and 
other  raters  and  generate  in  them  a  degree  of  enthusiasm  and  dedication  to  the  project. 


The  briefing  is  followed  by  1  1/2  hours  of  intensive  rater  training.  The  purposes  of  the 
rater  training  are  threefold:  (a)  to  familiarize  raters  with  the  supplemental  questionnaires  and 
the  four  rating  forms,  including  the  various  rating  scales,  scale  anchors,  and  item  formats;  (b) 
to  educate  raters  on  how  to  make  accurate  ratings;  and  (c)  to  provide  an  opportunity  for  practice 
on  two  of  the  rating  forms.  A  substantial  portion  of  the  training  consists  of  lecture,  with 
slides  used  as  learning  aids.  The  program  also  provides  opportunities  for  trainee  participation 
and  group  discussion.  Due  to  the  length  of  the  training  session,  it  is  advised  that  at  least  one 
break  be  given. 

The  third  part  of  the  rater  training  session  is  the  administration  of  the  rating  forms.  Test 
administrators  distribute  answer  sheets  to  appropriate  raters.  Each  incumbent  rates 
himself/herself  and  receives  ratings  from  his/her  supervisor  and  one  to  three  coworkers.  It  is 
the  responsibility  of  the  test  administrators  to  fill  in  the  names  and  social  security  account 
numbers  of  the  rater  and  ratee  on  each  answer  sheet  prior  to  distribution. 

The  trainer  should  spend  several  hours  conducting  an  entire  rater  training  session  for  test 
administrators,  beginning  with  the  briefing.  This  gives  test  administrators  the  opportunity  to 
experience  the  training  from  an  Incumbent's  viewpoint  and  also  to  observe  how  an  effective 
training  session  is  conducted.  The  trainer  should  encourage  administrators  to  voice  any  concerns 
or  questions  they  might  have.  The  trainer  should  then  provide  time  for  the  administrators  to 
familiarize  themselves  with  the  training  session.  As  was  the  case  for  the  Base  Introductory 
Briefing,  only  one  team  member  need  be  responsible  for  conducting  a  rater  training  session; 
however,  all  administrators  should  be  prepared. 

Appendices  E  and  F  contain  the  Administrator's  Guide  and  the  Trainee  Booklet  needed  to 
conduct  the  rater  training  session. 


Rater  Training  Briefing 


SLIDE  »1  -  JOB  PERFORMANCE  MEASUREMENT  PROJECT 

GOOD  MORNING/AFTERNOON.  MY  NAME  IS  _ .  THESE  ARE  MY  FELLOW  TEAM 

MEMBERS _  ANO  _ .  WE  REPRESENT  THE  AIR  FORCE  HUMAN 

RESOURCES  LABORATORY  AT  BROOKS  AFB,  TX.  WE  ARE  HERE  AS  PART  OF  THE  AIR  FORCE'S  JOB  PERFORMANCE 
MEASUREMENT  PROJECT.  I'D  LIKE  TO  TAKE  A  FEW  MINUTES  TO  GIVE  YOU  SOME  DETAIL  ON  THE  PROJECT  AND 
TO  TELL  YOU  WHAT  WE  'wlLL  BE  DOING  DURING  OUR  VISIT  ANO  HOW  YOU  WILL  BE  INVOLVED.  AFTER  MY 
BRIEFING,  WE  WILL  SPEND  ABOUT  1  1/2  HOURS  GOING  THROUGH  A  RATER  TRAINING  PROGRAM.  FOLLOWING  THE 
TRAINING,  EACH  OF  YOU  WILL  COMPLETE  A  SET  OF  RATING  FORMS,  RATING  EITHER  YOURSELF,  YOUR 
COWORKERS,  OR  YOUR  SUBORDINATES. 
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SLIDE  #2  -  OVERVIEW 


FIRST,  1*0  LIKE  TO  DISCUSS  SOME  REASONS  WHY  THE  AIR  FORCE  IS  CONDUCTING  THIS  PROJECT.  THE 
WORK  THE  AIR  FORCE  HAS  DONE  OVER  THE  LAST  YEAR  IN  YOUR  SPECIALTY  INVOLVES  THE  DEVELOPMENT  OF 
VARIOUS  WAYS  TO  MEASURE  AN  INOIVIOUAL'S  ABILITY  TO  PERFORM  HIS/HER  JOB.  I  WILL  THEN  DISCUSS  OUR 
PLANS  TO  COLLECT  JOB  PERFORMANCE  INFORMATION  WHILE  WE  ARE  HERE,  AND  THE  PAYOFFS  OF  THIS  WORK  FOR 
THE  AIR  FORCE. 


SLIDE  *3  -  BACKGROUND 

WHY  IS  THE  AIR  FORCE  DOING  THIS  WORK?  THERE  ARE  SEVERAL  MAIN  REASONS.  FIRST,  CONGRESS  HAS 
ASKED  THE  SERVICES  TO  REEXAMINE  THE  WAY  PEOPLE  ARE  SELECTED  AND  ASSIGNED  TO  SPECIALTIES,  AND 
ULTIMATELY,  HOW  THEY  ARE  PERFORMING  ON  THE  JOB.  THE  AIR  FORCE  IS  ALSO  INTERESTED  IN  ANSWERING 
THIS  QUESTION.  AS  WELL  AS  IN  USING  JOB  PERFORMANCE  INFORMATION  TO  KNOW  WHO  TO  TRAIN  AND  WHAT  TYPE 
OF  TRAINING  TO  GIVE. 


SLIDE  #4  -  OBJECTIVES 

THEREFORE,  THE  PRIMARY  OBJECTIVES  OF  OUR  EFFORT  ARE  TO  DEVELOP  A  PERFORMANCE  MEASUREMENT 
TECHNOLOGY  (A)  TO  EVALUATE  TRAINING  PROGRAMS,  AND  (B)  TO  ASSIST  IN  EVALUATING  THE  SETTING  OF 
ENLISTMENT  STANOAROS. 


SLIDE  #5  -  JOB  PERFORMANCE  MEASUREMENT  SYSTEM 

THE  JOB  PERFORMANCE  MEASUREMENT  SYSTEM  THAT  HAS  BEEN  DEVELOPED  FOR  YOUR  SPECIALTY  CONSISTS  OF 
THREE  MIN  COMPONENTS: 

1.  WALK-THROUGH  PERFORMANCE  TESTING 

2.  RATING  FORMS 

3.  FACTORS  RELATED  TO  PERFORMANCE  MEASUREMENT  QUESTIONNAIRES 

WALK-THROUGH  PERFORMANCE  TESTING  CONSISTS  OF  TWO  MAJOR  COMPONENTS:  (A)  HANDS-ON  TESTING,  AND 
(B)  INTERVIEW  TESTING.  HANOS-ON  TESTING  REQUIRES  INCUMBENTS  TO  PERFORM  TASKS  THEY  ACTUALLY  DO  ON 
THE  JOB.  INTERVIEW  TESTING  REQUIRES  INCUMBENTS  TO  EXPLAIN  THE  STEPS  INVOLVED  IN  PERFORMING  THE 
TASKS  THEY  ACTUALLY  DO  ON  THE  JOB. 

FOUR  DIFFERENT  TYPES  OF  RATING  FORMS  WERE  DEVELOPED.  THESE  RATING  FORMS  WILL  BE  COMPLETED  BY 
SUPERVISORS,  COWORKERS,  ANO  INCUMBENTS.  THESE  RATING  FORMS  ARE: 

1.  TASK  RATING  FORM 

2.  DIMENSIONAL  RATING  FORM 

3.  GLOBAL  RATING  FORM 

4.  AIR  FORCE-WIDE  RATING  FORM 

THROUGH  QUESTIONNAIRES  ADMINISTERED  AT  THE  SAME  TIME  AS  THE  RATING  FORMS,  WE  WILL  ALSO  BE 
COLLECTING  INFORMATION  ABOUT  AN  INCUMBENT'S  EXPERIENCE  ON  EQUIPMENT  ANO  SPECIFIC  TASKS,  PRIOR 
TRAINING  HE/SHE  HAS  RECEIVEO,  AND  LEVEL  OF  MOTIVATION. 
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SLIDE  #6  -  DATA  COLLECTION 

WE  ARE  CURRENTLY  IN  THE  MIDST  OF  FULL-SCALE  DATA  COLLECTION.  FOR  THE  NEXT  3  TO  4  MONTHS,  WE 
WILL  BE  COLLECTING  OATA  FROM  FIRST-TERMERS  IN  THE  426X2  SPECIALTY.  WE  WILL  VISIT  3-4  BASES  PER 
ENGINE  TYPE  (J-57,  J-79,  ANO  TF-33),  FOR  A  TOTAL  OF  9-12  BASES.  WE  WILL  UTILIZE  THREE  TEAMS  OF 
THREE  TEST  ADMINISTRATORS  EACH  (ONE  TEAM  PER  ENGINE  TYPE).  EACH  TEAM  WILL  SPEND  APPROXIMATELY  3 
WEEKS  AT  EACH  8ASE  ANO  WILL  EVALUATE  AT  LEAST  TWO  AIRMEN  PER  DAY.  WE  HOPE  TO  COLLECT  OATA  FROM  A 
TOTAL  OF  100-120  AIRMEN  PER  ENGINE,  USING  WALK-THROUGH  PERFORMANCE  TESTING,  RATING  FORMS,  ANO 
OTHER  QUESTIONNAIRES.  THIS  MEANS  WE  WILL  BE  TESTING  25-40  AIRMEN  PER  BASE  WHO  HAVE  BEEN  ON  THE 
JOB  APPROXIMATELY  6  MONTHS  AND  HAVE  UP  TO  48  MONTHS  OF  SERVICE  TIME. 


SLIDE  >7  -  DATA  COLLECTION  TEST  SCHEDULE 

THIS  IS  WHAT  OUR  TESTING  SCHEDULE  LOOKS  LIKE  FOR  YOUR  BASE  AND  OTHERS  INVOLVED  IN  THE  DATA 
COLLECTION  EFFORT.  (NOTE  TO  TRAINER:  TALK-THROUGH  THIS  SCHEDULE,  STEP  BY  STEP.) 


SLIDE  #8  -  AIR  FORCE  PAYOFFS 

WHAT  WILL  THIS  WORK  ACCOMPLISH?  IT  WILL: 

1.  IMPROVE  THE  MEASUREMENT  OF  PERFORMANCE. 

2.  IMPROVE  THE  AIR  FORCE'S  SYSTEM  OF  SELECTING,  CLASSIFYING,  AND  TRAINING 
INDIVIDUALS. 

3.  INCREASE  MISSION  READINESS. 
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Slides  #1, 
to  accommodate 
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Rater  Training  Briefing  Slides 


5,  6,  and  7  are  AFS-specif ic.  The  trainer  should  tailor  these  slides  as  necessary 
the  AfS  under  consideration. 
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V.  TRAINING  PROGRAM  VARIATIONS  AND  CONSTRAINTS 


The  information  included  in  this  training  package  is  designed  to  improve  the  performance  of 
test  administrators;  however,  there  is  no  way  to  guarantee  success  just  by  following  the  outline 
provided.  Many  equipment,  time,  situational,  and  personnel  constraints  can  affect  both  the 
training  approach  taken  and  the  outcomes  derived.  Therefore,  the  trainer  must  remain  flexible  in 
terms  of  materials  and  content  used.  Although  there  it  no  way  to  anticipate  all  possible 
situations,  it  is  possible  to  note  several  that  were  enc.  untered  in  the  Jet  Engine  Mechanic 
project,  as  well  as  other  issues  that  were  of  concern. 

In  most  situations,  WTPT  training  will  face  time  constraints.  For  AFS  426X2,  3  to  4  months 
of  training  time  was  available,  but  this  will  likely  prove  to  be  an  exception.  The  training 
program  developer  will  probably  be  allowed  no  more  than  several  weeks  to  bring  administrators  to 
the  desired  level  of  proficiency.  Consequently,  decisions  will  have  to  be  made  concerning 
optimization  of  training  content  and  format.  In  addition,  training  time  will  be  influenced  by 
the  characteristics  and  experience  of  the  test  administrators.  In  this  instance,  all  test 
administrators  were  former  Jet  Engine  Mechanics,  and  much  of  the  fundamental  training  could  be 
bypassed.  This  will  not  always  be  the  case. 

Subsequent  to  Jet  Engine  Mechanic  training/data  collection,  test  administrators  (in  most 
AFSs)  have  been  active-duty  NCOS.  For  active-duty  test  administrators,  their  total  time 
available  for  training  and  data  collection  is  limited  by  how  long  the  Major  Conmands  will  allow 
them  to  be  absent  from  their  units.  As  a  result,  no  more  than  2  weeks  were  available  for 
training  purposes,  and  all  training  was  encapsulated  within  this  period  of  time.  An  example  of 
this  training  schedule  can  be  found  in  Appendix  G.  Also,  when  active-duty  test  administrators 
are  used,  there  is  no  guarantee  that  those  used  during  pretest  will  return  for  data  collection. 
Consequently,  training  must  be  thorough  and  complete  on  each  occasion. 

Other  training  content  variations  should  be  noted  at  this  point.  First,  because  test 
administrators  remain  in  the  field  collecting  data  for  several  months,  there  is  no  guarantee  that 
high  levels  of  accuracy  and  reliability  will  be  maintained  across  test  administrators.  Because 
of  this  concern  in  post-Jet  Engine  Mechanic  data  collection  efforts,  a  shadow  scoring/ 
recalibration  training  program  was  Initiated.  This  approach  involved  periodic  "retraining"  of 
all  test  administrators.  Recalibration  training  focused  on  re-use  of  videotape  viewing 
subsequent  to  the  first  data  collection  trip.  In  addition,  throughout  data  collection,  a  shadow 
scoring  procedure  was  initiated,  whereby  multiple  test  administrators  rated  the  same  individuals, 
and  then  met,  discussed,  and  reached  agreement  on  what  ratings  should  have  been  given  to  each 
WTPT  examinee.  A  description  of  the  approach  used  for  shadow  scoring  is  Included  in  Appendix  H. 

The  Rater  Training  Session  presents  another  time-related  constraint.  Collection  of  rating 
forms  and  questionnaire  data  from  Incumbents,  coworkers,  and  supervisors  requires  a  considerable 
amount  of  time  to  accomplish  (as  much  as  2  hours).  For  the  Jet  Engine  Mechanic  Specialty,  it  was 
not  deemed  possible  to  remove  large  groups  of  mechanics  from  their  jobs  for  more  than  4  hours. 
This  served  to  limit  the  time  available  for  rater  training.  Therefore,  a  concise,  yet 
comprehensive  training  package  had  to  be  developed  to  overcome  this  time  constraint.  In 
addition,  test  administrators  had  to  be  trained  to  effectively  conduct  the  Rater  Training  Session 
within  prescribed  time  limits. 

Other  issues  related  to  rater  training  also  need  to  be  discussed.  First,  while  the  training 
exercises  presented  in  this  manual  are  for  Jet  Engine  Mechanics,  as  new  AFSs  are  tested,  new 
scenarios  for  Exercises  I  and  II  must  be  developed.  An  example  of  AFS  variations  is  included  in 
Appendix  I.  A  second  rater  training  variation  Involved  Exercise  III,  which  uses  a  discussion 
between  supervisors  to  highlight  rating  errors  frequently  made  by  raters,  with  AFS  426X2,  raters 


read  the  discussion  individually  prior  to  discussion.  In  subsequent  specialties,  individuals 
were  asked  to  “play  the  part*  of  supervisors  in  the  exercise,  reading  them  aloud.  This  was  found 
to  improve  group  participation  and  enhance  the  effectiveness  of  the  exercise. 

Many  other  variations  and  constraints  of  varying  importance  can  be  noted,  ranging  from 
personality  conflicts  to  turnover  issues.  The  issues  presented  above  are  some  of  the  most 
important,  but  the  training  program  developer  would  be  wise  to  anticipate  potential  problems  and 
devise  effective  strategies  to  deal  with  them. 


VI.  CONCLUDING  REMARKS 

This  training  package  has  been  assembled  as  an  example  of  the  type  of  training  given  AFS 
426X2  test  administrators.  It  has  also  been  designed  to  serve  as  a  model  for  future  training 
efforts  of  this  sort.  Because  of  the  requirements  inherent  in  the  Air  Force's  criterion 

development  effort,  the  focus  of  training  has  been  twofold.  Included  are  descriptions  and 

materials  necessary  to  train  (a)  WTPT  administrators  to  provide  accurate  assessments  of  the 
performance  of  first-term  airmen,  and  (b)  rating  form  administrators  to  provide  rater  training. 
A  list  of  pertinent  references  used  in  the  development  of  this  training  package  is  included  at 
the  end  of  the  package;  however,  the  reader  interested  in  additional  information  about  WTPT  or 

rating  form  development  is  specifically  referred  to  Alba  and  Dickinson  (1985)  and  Bierstedt 

(1985). 


BIBLIOGRAPHY 


Alba,  P.A.,  &  Dickinson,  T.L.  (1985).  Walk-through  performance  test  development  for  jet  engine 
mechanic  (APS  426X2).  Unpublished  manuscript.  Brooks  APB,  TX:  Air  Force  Human  Resources 
Laboratory. 

Alba,  P.A.,  l>  Wilcox,  T.  (  1985).  Walk-through  performance  testing  administrator's  manual  for  jet 
engine  mechanic  (APS  426X2).  Unpublished  manuscript.  Brooks  APB,  TX:  Air  Force  human 
Resources  Laboratory. 

Bierstedt,  S.A.  (1985).  Rating  form  development  for  jet  engine  mechanic  specialty.  Unpublished 
manuscript.  Brooks  APB,  TX:  Air  Force  Human  Resources  Laboratory. 

Campbell,  D.T.  (1958).  Systematic  error  on  the  part  of  human  links  in  communication  systems. 
Information  and  Control.  J,  334-369. 

Campbell,  R.C.  (1979,  October).  Hands-On  testing:  A  mode)  for  scorer  training.  Paper  presented 
at  the  21st  Annual  Meeting  of  the  Military  Testing  Association,  San  Diego,  CA. 

Carrol,  S.J.,  &  Schneier,  C.E.  (1982).  Performance  appraisal  and  review  systems:  The 

Identification,  measurement,  and  development  of  performance  in  organizations.  Glenview,  IL: 
Scott,  Foresman. 

Eitington,  J.  (1976).  Training  methods  manual.  Washington,  DC:  Department  of  the  Interior. 

Frick,  J.,  &  Semmel,  M.U.  (1978).  Observer  agreement  and  reliabilities  of  classroom 
observational  measures.  Review  of  Educational  Research.  48,  157-184. 

Goldstein,  A.P.,  l>  Sorcher,  M.  (1974).  Changing  supervisor  behavior.  New  York:  Pergamon. 

Hakel,  M.O.  (1976;.  Some  questions  and  comments  about  applied  learning.  Persona)  Psychology. 

29,  361-363. 

Hedge,  J.W.,  Lipscomb,  M.S.,  Ballentine,  R.O.,  &  Gould,  R.B.  (1984,  August).  Walk-Through 

performance  testing:  An  innovative  approach  to  work  sample  testing.  Symposium  held  at  the 
92nd  Annual  Convention  of  the  American  Psychological  Association,  Toronto,  Canada. 

Herbert,  J.,  &  Athredge,  C.  (1975).  A  guide  for  developers  and  users  of  observation  systems  and 
manuals.  American  Research  Journal.  12,  1-20. 

Kahn,  R.L.,  l>  Cannel,  C.F.,  (1983).  The  dynamics  of  interviewing:  Theory,  technique,  and 
cases.  New  York:  John  Wiley. 

Landy,  F.J.,  &  Farr,  J.L.  (1983).  The  measurement  of  work  performance:  Methods,  theory,  and 
applications.  New  York:  Academic  Press. 

Latham,  G.P.,  &  Wexley,  K.N.,  (1981).  Increasing  productivity  through  performance  appraisal. 
Reading,  MA:  Addlson-Wesley. 

National  Institute  on  Orug  Abuse  (1979).  Training  of  tralners-participant' s  manual.  Washington, 
0C:  Department  of  Health,  Education,  and  Welfare. 


National  Institute  on  Drug  Abuse  (1979).  Training  of  trainers-trainer1 s  manual.  Washington, 

OC:  Department  of  Health,  Education,  and  Welfare. 

Office  of  the  Assistant  Secretary  of  Defense  (MRAR)  (  1983,  December).  Second  annual  report  to 
congress  on  joint-service  efforts  to  link  standards  for  enlistment  to  on-the-job 
performance.  Washington,  DC:  Department  of  Defense. 

Sorcher,  M.,  &  Goldstein,  A.P.  (1972).  A  behavior  modeling  approach  in  training.  Personnel 
Administration,  35,  35-41.  ~ 

Spool,  M.O.  (1978).  Training  programs  for  observers  of  behavior:  A  review.  Personal 
Psychology.  31,  853-888. 

Stallings,  J.A.  (1977).  A  handbook  on  classroom  observations  and  teaching  models.  Belmont,  CA: 
Wadsworth  Publishing  Co.,  Inc. 

Thorton,  G.C.,  t  Zorich,  S.  (1980).  Training  to  improve  observer  accuracy.  Journal  of  Applied 
Psychology.  65.  351-351. 

United  States  Department  of  Labor  (1972).  Handbook  for  analyzing  jobs.  Washington,  DC:  U.S. 
Government  Printing  Office. 

University  of  Michigan  Survey  Research  Center  (1976).  Interviewer's  Manual  (rev.  ed.).  Ann 
Arbor,  MI:  Institute  for  Social  Research. 

Warwick,  D.C.,  &  Linninger,  C.A.  (1975).  The  sample  survey  theory  and  practice.  New  York: 
McGraw-Hill. 

Wexley,  K.N.,  t  Latham,  G.P.  (1981).  Developing  and  training  human  resources  in  organizations. 
Glenview,  IL:  Scott,  Foresman. 

Wexley,  K.N.,  Sanders,  R.E.,  t  Yukl,  G.A.  (1973).  Training  interviewers  to  eliminate  contrast 
effects  in  employment  interviews.  Journal  of  Applied  Psychology.  57,  233-236. 


APPENDIX  A:  SUGGESTED  AGENDA  FOR 
TRAINING  ORIENTATION  WORKSHOP 


Introduction 

Review  of  uPMS  Instruments 

A.  Administrator's  Manual 

B.  Work  Sample  Tests 

1.  Hands-on  Tests 

2.  Interview  Tests 

3.  Related  Technical  Orders/Job  Guides 

C.  Rating  Forms 

1.  Task 

2.  Dimensional 

3.  Global 

4.  Air  Force-Wide 

0.  Supplemental  Questionnaires 

1.  General  Acceptability/Utility 

2.  General  Background 

3.  Rating  Form  Questionnaire 

4.  Task  Experience  Ratings 

Observation  Training 

A.  Techniques  for  Accurate  Observation 

B.  Practice 


APPENDIX  B:  CHECKLIST  OF  WORKSHOP  MATERIALS 

Overhead  Projector 
Videocassette  Players 
Workshop  Introduction  Slides  and  Script 
Workshop  Agenda 

Overview  Briefing  Slides  &  Script 

Base  Introductory  Briefing  Slides  and  Script 

Administrator's  Manual  for  WTPT 

Interview  Training  Slides  and  Script 

Interviewer  Do's  and  Oon'ts  (Copies  of  Slides  #9  and  10) 

Interview  Modeling  Exercise  Videotape  6  Discussion  Guidelines 
WTPT  Test  Booklets 
Hands-On  Training  Slides  and  Script 
Videotapes  of  WTPT  Hands-On  Tasks 
Rater  Training  Briefing  Slides  and  Script 
Rater  Training  Administrator's  Guide 
Rater  Training  Exercise  I  and  II  Rating  Forms 
Rater  Training  Trainee  Booklet 
Sample  Rating  Form  Booklet 


These  Items  should  be  Included  In  the  administrator's  workshop  packets. 


APPENDIX  C:  RATER  TRAINING  CHECKLIST 


Supplies 

Overhead  Projector  _ 

Briefing  Notes  _ 

Briefing  Slides  _ 

Administrator' s  Guide  _ 

Training  Slides  _ 

Trainee  Booklets  _ 

Sample  Rating  Form  Booklets  _ 

Exercise  I  -  Air  Force-Wide  _ 

Exercise  II  -  Dimensional  _ 

Extra  Pens  _ 

Actions 

Pass  around  sign-up  sheet  _ 

Distribute  Trainee  Booklets  _ 

Give  Briefings  _ 

Distribute  Sample  Rating  Form  Booklets  _ 

Distribute  Exercise  I  -  Air  Force-Wide  _ 

Distribute  Exercise  II  -  Dimensional  _ 

Pick  up  Trainee  Booklets  _ 

Pick  up  Sample  Rating  Form  Booklets  _ 

Pick  up  Exercise  I  -  Air  Force-Wide  _ 

Pick  up  Exercise  II  -  Dimensional  _ 

Distribute  Privacy  Act  Statement  _ 

Things  to  Remember 

1.  Talk  to  your  audience  -  especially  about  those  portions  of  the  training  that  are  not  part  of 
the  Trainee  Booklet. 

2.  Make  sure  the  overhead  projector  is  properly  focused  --  check  for  this  after  you  finish  the 
briefing  and  are  ready  to  begin  the  actual  training. 

3.  Be  sure  to  give  at  least  one  break  during  the  training  session.  A  good  time  is  immediately 
prior  to  administering  the  rating  forms.  You  might  give  two  breaks  during  the  afternoon 
session. 

4.  Pick  up  all  booklets  and  exercises  —  take  a  quick  count  of  how  many  you  hand  out  and  how 
many  you  pick  up.  Every  booklet  must  be  accounted  for. 

5.  Review  each  rating  form  booklet  as  it  is  turned  in.  Ensure  there  are  no  missing  data. 

6.  This  is  the  first  time  the  incumbents  and  their  supervisors  and  peers  will  see  you  and  hear 
about  why  you  are  at  their  base.  Present  a  good  attitude.  Enthusiasm  is  caught,  not  taught:1. 


APPENDIX  D:  privacy  act  statement 


In  accordance  with  the  Air  Force  Privacy  Act  Program,  AFR  12-35,  paragraph  8,  the  following 
information  about  this  survey  is  provided: 

a.  Authority.  10  U.S.C.,  8012,  Secretary  of  the  Air  Force:  Powers  and  Duties  Delegation  by. 

b.  Principal  Purpose.  The  data  collected  are  to  be  used  for  research  purposes  only. 

c.  Routine  Use.  The  information  collected  will  be  used  by  the  Air  Force  Human  Resources 
Laboratory  in  a  research  study  to  link  job  performance  with  enlistment  standards. 

d.  Participation  in  this  survey  is  entirely  voluntary. 

e.  No  adverse  action  of  any  kind  may  be  taken  against  any  individual  who  elects  not  to 
participate  in  this  survey. 


i 
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APPENDIX  E:  ADMINISTRATOR’S  GUIDE 


HAND  OUT  RATER  TRAINING  BOOKLETS,  THEN  SAY:  HE  ARE  GOING  TO  BEGIN  THE  RATER  TRAINING  SESSION. 
PLEASE  POLLOH  ALONG  WHILE  1  READ  OUT  LOUD.  DO  NOT  MARK  IN  THE  BOOKLETS. 

I.  Introduction 

For  the  next  several  hours,  your  time  will  be  spent  focusing  on  the  rating  forms  you  will 
complete  as  part  of  this  research  project.  You  will  use  several  rating  forms  to  rate  the 
performance  of  your  coworkers  and  yourself.  In  addition,  your  supervisor  and  several  coworkers 
will  use  the  same  forms  to  rate  you. 

Before  you  use  any  of  the  rating  forms,  we  are  going  to  talk  about  each  form,  its  purpose,  and 
how  to  use  each  form  to  effectively  rate  a  coworker  or  yourself.  We  are  also  going  to  discuss 
some  ideas  that  will  help  you  use  the  rating  forms  and  make  the  most  accurate  ratings  possible. 

It  is  essential  to  the  outcome  of  this  project  that  you  be  truthful  and  honest  in  your  ratings. 
The  ratings  will  not  be  seen  by  your  coworkers,  supervisors,  or  anyone  else  connected  with  your 
unit.  The  data  collected  will  be  seen  only  by  Air  Force  Human  Resources  Laboratory  personnel  and 
the  private  contractor  associated  with  this  project.  The  information  you  provide  will  be  coded 
to  assure  anonymity,  and  the  rating  forms  will  subsequently  be  destroyed.  The  ratings  will  be 
used  for  research  purposes  only  and  will  in  no  way  affect  anyone’s  career.  Therefore,  please 
rate  each  person  as  accurately  as  possible. 

II.  Explanation  of  Rating  Scale 

A.  SHOW  OVERHEAD  OF  RATING  SCALE  -  PROFICIENCY  BASE 

Each  rating  form  uses  a  5-point  scale  (5  high  -  1  low).  To  help  you  make  more  accurate  ratings, 
the  points  on  the  scale  are  labeled,  or  anchored.  You  will  notice  that  these  anchors  describe  a 
certain  level  of  proficiency.  Proficiency  refers  to  how  skilled  a  person  is  at  performing 
various  tasks  on  the  job,  ignoring  interpersonal  factors  (willingness  to  work,  cooperating  with 
others)  or  situational  factors  (lack  of  tools  or  parts,  weather  conditions). 

READ  THROUGH  EACH  LEVEL 

B.  SHOW  OVERHEAD  OF  RATING  SCALE  -  PERFORMANCE  BASE 

One  of  the  rating  forms  you  will  be  using  is  not  technical  in  its  orientation.  The  scale  anchors 
of  this  form  refer  to  various  levels  of  performance  rather  than  levels  of  proficiency.  The 
rating  form  will  examine  not  only  technical  ability,  but  also  other  factors  that  contribute  to  a 
mechanic's  performance  on  the  job. 

READ  THROUGH  EACH  LEVEL 

C.  SHOW  OVERHEAD  OF  BEHAVIORAL  EXAMPLES  -  QUALITY  CONTROL 

Three  of  the  rating  forms  will  include  short  paragraphs  that  describe  behavior  typical  to  each 
level  on  the  scale.  These  behavioral  examples  will  aid  you  in  making  your  ratings. 

REFER  TO  THE  OVERHEAD  AND  SAY: 


THIS  PARTICULAR  ITEM  RATES  A  PERSON'S  LEVEL  OF  PROFICIENCY  IN  THE  AREA  OF 
NOTICE  THAT  A  DEFINITION  OF  QUALITY  CONTROL  IS  GIVEN  FIRST. 


DUALITY  CONTROL. 


POINT  TO  AND  READ  DEFINITION 


THE  DEFINITION  IS  FOLLOWED  BY  THE  ANCHOR  FOR  EACH  LEVEL.  THE  CORRESPONDING  NUMBER  FOR  EACH  LEVEL. 


AND  THE  BEHAV IORAL  EXAMPLES. 

POINT  TO  EXAMPLES,  READ  THROUGH  EACH  EXAMPLE,  BEGINNING  WITH  LEVEL  5.  READ  ACROSS  FORM,  FROM 
LEFT  TO  RIGHT,  STATING  THE  LEVEL.  THE  NUMBER,  AND  THEN  THE  BEHAVIORAL  EXAMPLE. 

D.  SHOW  OVERHEAD  OF  BEHAVIORAL  EXAMPLES  -  INITIATIVE/EFFORT 

REFER  TO  THE  OVERHEAD  AND  SAY: 

HERE  IS  ANOTHER  SET  OF  BEHAVIORAL  EXAMPLES.  THESE  EXAMPLES  REFER  TO  VARIOUS  LEVELS  OF 


PERFORMANCE  IN  THE  AREA  OF  INITIATIVE/EFFORT. 


ALLOW  SUFFICIENT  TIME  FOR  INCUMBENTS  TO  READ  SEVERAL  EXAMPLES. 

IT  IS  IMPORTANT  THAT  YOU  READ  ALL  THE  BEHAVIORAL  EXAMPLES  THOROUGHLY  BEFORE  DECIDING  ON  A 


RATING.  DO  NOT  EXPECT  A  PERSON'S  JOB-RELATED  BEHAVIOR  TO  BE  IDENTICAL  TO  A  GIVEN  BEHAVIORAL 


EXAMPLE.  THE  EXAMPLES  ARE  IN  NO  WAY  MEANT  TO  INCLUDE  ALL  POSSIBLE  BEHAVIORS.  USE  THE  EXAMPLES 


ONLY  AS  GUIDELINES  FOR  DETERMINING  A  RATING. 


ASK:  ARE  THERE  ANY  QUESTIONS  REGARDING  THE  RATING  SCALE? 


III.  Explanation  of  Rating  Forms 

DISTRIBUTE  SAMPLE  COPIES  OF  RATING  FORM  BOOKLETS.  SAY:  THESE  ARE  SAMPLES  OF  THE  RATING  FORM 
BOOKLETS  YOU  WILL  BE  USING  LATER.  WE  WILL  LOOK  AT  EACH  FORM  BRIEFLY.  DO  NOT  MARK  IN  THESE 


BOOKLETS.  PLEASE  TURN  TO  THE  GLOBAL  RATING  FORM  ON  PAGE  3. 


A.  GLOBAL  RATING  FORM 

This  rating  form  has  two  items.  The  first  item  asks  for  a  rating  of  a  person's  overall  technical 
proficiency.  The  second  item  asks  for  a  rating  of  a  person's  overall  social/interpersonal 
proficiency.  This  form  utilizes  behavioral  examples  to  aid  you  in  making  your  ratings. 

ALLOW  TIME  FOR  EXAMINATION  OF  THE  FORM.  THEN  SAY:  PLEASE  TURN  TO  THE  DIMENSIONAL  RATING  FORM  ON 
PAGE  6. 

B.  DIMENSIONAL  RATING  FORM 

The  purpose  of  this  rating  form  Is  to  evaluate  a  mechanic's  proficiency  on  a  number  of  important 
job  areas  or  dimensions.  Again,  the  behavioral  examples  will  serve  as  a  guide  in  determining 
your  ratings. 

ALLOW  TIME  FOR  EXAMINATION  OF  THE  FORM.  THEN  SAY:  PLEASE  TURN  TO  THE  TASK  RATING  FORM  ON  PAGE  13. 

C.  TASK  RATING  FORMS 


The  purpose  of  this  rating  form  is  to  evaluate  proficiency  on  a  variety  of  tasks  critical  to 
first-term  Jet  Engine  Mechanics.  This  form  contains  tasks  common  to  the  J-57,  J-79,  and  TF-33 


engines  and  to  the  flightline  and  shop  areas.  It  also  contains  items  unique  to  the  engine  you 
work  on  and  your  functional  area. 


ALLOW  TIME  FOR  EXAMINATION  OF  THE  FORM.  THEN  SAT :  PLEASE  TURN  TO  THE  AIR  FORCE-WIDE  RATING  FORM 
ON  PAGE  16. 

0.  AIR  FORCE-WIDE  RATING  FORM 

This  rating  form  aoes  not  evaluate  technical  job  skill.  It  rates  a  person  on  elements  important 
to  overall  success  in  the  Air  Force.  This  is  the  rating  form  mentioned  previously  that  uses 
performance  rather  than  proficiency  as  a  basis  for  rating  scale  anchors.  Again,  you  will  utilize 
behavioral  examples  in  making  your  ratings. 

E.  SUPPLEMENTAL  QUESTIONNAIRES 

You  will  notice  that  there  are  several  other  questionnaires  included  in  the  booklet.  On  pages 
1-2  is  a  General  Background  questionnaire  which  asks  for  information  on  yourself  and  the  person 
you  are  rating.  The  Rating  Form  Questionnaire,  found  on  pages  25-26,  asks  you  to  give  an 
evaluation  of  the  rating  forms.  The  final  item  in  the  booklet  is  the  Task  Experience  Ratings 
Questionnaire,  which  requires  you  to  provide  information  as  to  how  much  experience  you've  had  on 
various  tasks.  You  will  complete  this  questionnaire  only  on  yourself,  not  on  your  peers, 
supervisors,  or  subordinates. 

SAY:  ARE  THERE  ANY  QUESTIONS  REGAROING  THE  RATING  FORMS? 


AFTER  ALL  QUESTIONS  HAVE  BEEN  ANSWERED,  COLLECT  THE  RATING  FORM  BOOKLETS.  WHEN  ALL  BOOKLETS  HAVE 
BEEN  COLLECTED,  SAY:  NOW  TURN  TO  SECTION  IV  IN  YOUR  TRAINING  BOOKLET. 

IV.  Tips  on  Making  Accurate  Ratings 

A.  GENERAL  INFORMATION 

The  most  important  thing  to  remember  when  making  your  ratings  is  to  focus  your  attention  only  on 
the  person  you  are  rating,  and  only  on  the  person's  ability  to  perform.  Avoid  comparing  the 
person  with  coworkers  you've  rated  previously  or  those  you  will  subsequently  evaluate. 

Remember  that  even  though  you  and  the  people  you  rate  are  similar  because  you  are  all  Jet  Engine 
Mechanics,  each  person  has  a  unique  set  of  strengths  and  weaknesses.  Also,  be  aware  that  poor  or 
outstanding  performance  in  one  area  does  not  dictate  the  quality  of  performance  in  other  areas. 
Rate  the  person  according  to  the  individual's  ability  to  perform  on  the  job,  and  focus  on 
observable  behavior.  Finally,  do  not  be  afraid  to  use  the  entire  range  of  the  scale  when 

appropriate.  Your  honesty  will  serve  to  ensure  the  accuracy  of  your  ratings. 

B.  EXERCISE 

The  following  is  a  conversation  between  several  engine  crew  chiefs.  Read  it  and  see  if  you  can 
Identify  factors  that  might  contribute  to  Inaccurate  ratings. 

SAY:  TAKE  A  FEW  MINUTES  TO  READ  THIS  CONVERSATION.  WHEN  YOU  ARE  FINISHED.  WE  WILL  DISCUSS  IT. 
Andrews:  Here  we  are  again  at  our  weekly  gripe  session. 

Baker:  Yeah.  Seems  like  we  never  run  out  of  problems  to  complain  about. 


Cullen:  I  don't  know  why  you  guys  have  so  many  problems  on  your  crew. 
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Baker: 

Cul len: 

Andrews: 

Cullen: 


Andrews : 

Baker: 

Davis: 


Andrews : 

Cullen: 

Andrews: 


Oavls: 


Because  we  don't  have  the  cream  of  the  crop  that  you  have. 

You've  got  the  same  kind  of  guys  working  for  you  that  I  have.  Just  think  about 
it.  All  the  airmen  take  tests  to  identify  who  has  the  aptitude  for  what.  Right? 

R  ight. 

OK.  So  you've  got  a  bunch  of  guys  with  mechanical  aptitude.  These  guys  all 
receive  the  same  training  at  tech  school  to  learn  to  be  competent  mechanics.  They 
all  come  out  of  tech  school  with  the  same  training.  That's  the  purpose  of  tech 
school.  So  there  shouldn't  be  anyone  on  your  crew  who  is  more  qualified  to  do  the 
job  than  someone  else.  Granted,  if  you  take  any  one  of  the  airmen  and  compare  him 
to  the  average  guy  on  the  street,  sure,  he  is  going  to  look  like  a  mechanical 
genius.  But  among  his  fellow  crew  members,  he's  just  another  average  guy  like  all 
the  rest. 

I  disagree.  On  my  crew.  I've  got  a  couple  of  guys  who  are  just  outstanding.  Next 
to  these  guys,  the  rest  of  my  airmen  are  way  below  average. 

I  know  what  you  mean.  I  received  a  new  recruit  2  weeks  ago.  You  wouldn't  believe 
how  badly  he  has  performed  in  the  short  time  he's  been  here.  I  guess  I've  got  a 
real  loser  on  my  hands.  He'd  be  right  at  home  with  your  guys,  Andrews. 

I  guess  I  sort  of  understand  what  you  are  talking  about,  but  on  the  other  side  of 
the  coin.  The  last  recruit  that  came  onto  my  crew  was  fresh  out  of  tech  school. 
He  had  been  here  only  a  week  or  so  and  I  gave  him  AFTO  Form  349  to  complete.  Well, 
he  completed  the  form  without  even  asking  a  question  and  used  the  06  Code  Manual 
like  a  pro.  I  figure  I  have  a  super  mechanic  in  this  guy.  I  mean,  if  he  can  do 
that,  he  can  do  just  about  anything  I  assign  him  to  do. 

You  know,  Cullen,  I've  heard  how  difficult  you  can  be  when  you  rate  your  airmen. 
I've  heard  some  of  your  people  complain  that  no  matter  how  hard  they  try,  you  won't 
give  them  more  than  an  average  rating.  They  say  they  feel  like  they're  banging 
their  heads  aginst  a  brick  wall. 

I  have  a  pretty  good  idea  who  you're  talking  about.  If  they  think  it's  hard  to  get 
a  good  rating  out  of  me  now,  just  wait  until  I  rate  them  next  time.  They'll  find 
out  how  hard  a  brick  wall  really  is. 

I  guess  I  don't  think  like  you  guys  at  all.  I  try  to  be  a  friend  to  my  airmen. 
Some  of  them  are  away  from  home  for  the  first  time  and  I  don't  like  to  make  it  any 
rougher  on  them  than  I  need  to.  I  try  to  give  my  airmen  the  benefit  of  the  doubt 
when  I'm  rating  them.  You'd  be  surprised  what  you  learn  about  your  airmen  if  you 
try  to  be  their  friend.  For  Instance,  there's  a  guy  on  my  crew  now  who  reminds  me 
of  myself  when  I  was  his  age.  He's  from  a  small  town  and  joined  the  Air  Force  to 
get  away  from  that  atmosphere  and  see  a  little  of  the  world.  I  can  really  identify 
with  him. 


Well,  I  don't  know  about  being  buddy-buddy  with  my  airmen,  but  I  do  know  that  right 
now  everyone  in  my  unit  is  doing  a  good  job  as  far  as  I'm  concerned. 


Andrews:  Why  is  that? 


Davis: 


well,  I'm  up  for  promotion  in  a  couple  of  months.  Ana  let's  face  it,  the  better  my 
airmen  look  on  their  ratings,  the  better  I  look  as  a  supervisor.  A no  the  better  I 
look,  the  better  my  chances  are  for  promotion! 

WHEN  IT  APPEARS  THAT  EVERYONE  HAS  REAO  THE  CONVERSATION,  BEGIN  THE  OISCUSSION.  SAY:  THE  THIRD 
TIME  CULLEN  SPEAKS.  HE  TALKS  ABOUT  HOW  EVERY  MEMBER  OP  HIS  CREW  HAS  THE  SAME  TRAINING  AND 
[UALIFICATIONS.  HIS  WAY  OF  THINKING  DIRECTS  HIM  TO  THE  INCORRECT  CONCLUSION  THAT  EVERYONE  IS 
AVERAGE.  HE  IS  LIKELY  TO  GIVE  ONLY  AVERAGE  PERFORMANCE  RATINGS  TO  HIS  CREW  MEMBERS,  AND  WILL 
PROBABLY  NOT  RECOGNIZE  OUTSTANDING  OR  EVEN  BELOW-AVERAGE  PERFORMANCE. 

ANDREWS  MAKES  A  MISTAKE  IN  THE  NEXT  STATEMENT  BY  COMPARING  CREW  MEMBERS  WITH  EACH  OTHER.  THE 
LESSON  HERE  IS  TO  RATE  EACH  PERSON  ACCORDING  TO  HIS/HER  OWN  ABILITY  TO  DO  THE  JOB. 

BAKER  FOLLOWS  BY  MAKING  A  JUDGMENT  THAT  A  CREW  MEMBER  IS  "A  REAL  LOSER"  WHEN  HE'S  ONLY  8EEN  ON 
THE  CREW  A  COUPLE  OF  WEEKS.  WHEN  YOU  MAKE  YOUR  RATINGS.  BE  SURE  TO  BASE  THEM  ON  A  NUMBER  OF 
OBSERVATIONS  OF  A  PERSON'S  PERFORMANCE  AND  NOT  JUST  ON  ONE  INCIDENT. 

DAVIS  MAKES  THE  NEXT  MISTAKE  WHEN  HE  ASSUMES  THAT  IF  A  CREW  MEMBER  CAN  DO  ONE  TASK  WELL,  HE  CAN 
DO  EVERYTHING  WELL.  WHEN  YOU  MAKE  YOUR  RATINGS.  TRY  NOT  TO  MAKE  THIS  TYPE  OF  MISLEADING 
ASSUMPTION.  IN  OTHER  WORDS.  JUST  BECAUSE  A  PERSON  GETS  A  HIGH  RATING  ON  ONE  TASK  OR  DIMENSION, 
THAT  DOESN'T  MEAN  THE  PERSON  WILL  ALSO  GET  HIGH  RATINGS  ON  THE  OTHER  TASKS  OR  DIMENSIONS.  THE 
SAME  APPLIES  FOR  LOW  RATINGS. 

ANDREWS  STATES  THAT  CULLEN  HAS  A  REPUTATION  OF  BEING  AN  UNREASONABLY  TOUGH  RATER,  AND  IN  THE  NEXT 
STATEMENT,  CULLEN  CONFIRMS  THIS  FACT.  CULLEN  IS  NOT  BASING  HIS  RATINGS  ON  OBSERVED  BEHAVIOR. 
INSTEAD,  HE  IS  USING  THE  LOW  RATINGS  AS  A  WAY  TO  GET  REVENGE  ON  HIS  CREW  MEMBERS.  SOME  PEOPLE 
MIGHT  USE  HIGH  RATINGS  TO  REWARD  OTHERS.  EITHER  MAY  IS  UNFAIR.  THE  RULE  HERE  IS  TO  BASE  YOUR 
RATINGS  ON  OBSERVED  BEHAVIOR. 

IN  THE  NEXT  STATEMENT.  ANDREWS  INDICATES  THAT  HE  IS  A  LENIENT.  OR  EASY,  RATER.  ON  TOP  OF  THAT, 
ANDREWS  IS  LIKELY  TO  RATE  A  PERSON  HIGHER  IF  THAT  PERSON  IS  SOMEHOW  SIMILAR  TO  HIMSELF  IN 
BACKGROUND,  INTERESTS.  ETC.  AGAIN,  BASE  YOUR  RATINGS  ON  OBSERVED  BEHAVIOR. 

FINALLY.  IN  THE  LAST  STATEMENT.  DAVIS  IS  ASSUMING  THAT  HIS  OWN  WORTH  AS  A  SUPERVISOR  IS  ENHANCED 
BY  THE  FAVORABLE  RATINGS  HE  GIVES  HIS  CREW  MEMBERS.  HE  WOULD  BE  UNLIKE1 Y  TO  GIVE  LOW  RATINGS 
BECAUSE  THAT  WOULD  MAKE  HIM  LOOK  BAD  AS  A  SUPERVISOR.  THE  RATINGS  YOU  GIVE  WILL  IN  NO  WAY 
REFLECT  ON  YOU. 

IN  SUMMARY.  THE  ONE  MOST  IMPORTANT  THING  TO  REMEMBER  WHEN  MAKING  YOUR  RATINGS  IS  TO  FOCUS  ON 


HAND  OUT  EXERCISE  -  AIR  FORCE-WIDE  RATING  FORM. 

THEN  SAY:  READ  THE  STORY  AND  COMPLETE  THE  RATING  FORM.  WE  WILL  DISCUSS  YOUR  RATINGS  WHEN 
EVERYONE  IS  FINISHED.  PLEASE  GO  THROUGH  THE  ENTIRE  FORM. 


EXERCISE  I 

AIR  FORCE-WIDE  RATING  FORM 


♦NOTE  TO  THE  ADMINISTRATORS: 

The  underlined  statements  in  Exercise  I  and  II  are  justifications  for  ratings.  The  parenthetical 
notation  following  each  statement  refers  to  the  dimension/performance  factor  number  and  its 
appropriate  rating. 

Airman  Martin,  an  engine  crew  chief,  was  in  the  process  of  reassembling  his  engine  when  the 
shop  chief  informed  him  that  the  engine  had  to  be  ready  for  final  inspection  by  Friday. 


Airman  Martin  said  that  there  would  be  no  problem  meeting  that  deadline,  and  he  quickly  took 
charge  of  the  situation,  gaining  the  support  of  his  subordinates  (5-5).  Using  his  technical 
knowledge  and  skills,  he  finished  the  assembly  late  Thursday  afternoon  but  reguired  supervision 
during  the  buildup  of  the  next  section  (1-3). 

Prior  to  leaving,  it  was  discovered  that  a  tool  was  missing.  Being  a  truthful  individual, 
and  knowing  the  Importance  of  the  situation.  Airman  Martin  informed  the  shop  chief  immediately 
(4-5).  Staying  calm  and  in  control,  he  volunteered,  without  hesitation,  to  stay  until  the  tool 
was  found  (2-5). 

The  next  morning.  Airmen  Martin  was  Informed  to  report  to  the  Branch  Chief  at  1300.  At  1300, 
he  reported  to  the  Branch  Office  (3-4).  The  Branch  Chief  called  him  in  and  commended  him  on  a 
job  well  done  and  on  his  self-control  in  the  situation  of  the  missing  tool  (8-5).  Also,  the 
Branch  Chief  said  his  appearance  and  military  bearing  were  outstanding  (6-5)  and  to  keep  up  the 
good  work. 

After  getting  off  duty  later  that  day.  Airman  Martin  went  to  the  Tech  Order  room  to  check  out 
some  T.O.s  to  study  the  problem  on  the  engine  he  was  getting  next  (7-4). 

WHEN  IT  APPEARS  THAT  EVERYONE  HAS  COMPLETED  THE  FORM,  SAY:  THERE  ARE  KEY  STATEMENTS  IN  THIS 
STORY  THAT  INOICATE  THE  RATING  THAT  AIRMAN  MARTIN  SHOULD  RECEIVE  ON  EACH  PERFORMANCE  FACTOR  OF 
THE  AIR  FORCE-WIDE  RATING  FORM.  LET'S  GO  THROUGH  THE  RATING  FORM  ONE  FACTOR  AT  A  TIME  AND 
OISCUSS  THE  RATINGS. 

WHAT  RATING  DID  YOU  GIVE  AIRMAN  MARTIN  ON  PERFORMANCE  FACTOR  1:  TECHNICAL  KNOWLEDGE/SKILL? 

ALLOW  TIME  FOR  MEMBERS  OF  THE  GROUP  TO  ANSWER,  THEN  SHOW  OVERHEAD  OF  FACTOR  1  AND  SAY:  THE 
APPROPRIATE  RATING  WOULD  BE  A  3  ON  THIS  FACTOR  BECAUSE  THE  SECOND  PARAGRAPH  STATES  THAT  AIRMAN 
MARTIN  NEEDED  SUPERVISION  TO  COMPLETE  A  DIFFICULT  PART  OF  HIS  TASK.  THAT  STATEMENT  CORRESPONDS 
TO  THE  BEHAVORIAL  EXAMPLE  OF  LEVEL  3.  ANY  QUESTIONS? 

CONTINUE  WITH  EACH  PERFORMANCE  FACTOR  IN  THE  FOLLOWING  MANNER: 


ASK  THE  GROUP  FOR  THEIR  RATINGS 


SHOW  THE  APPROPRIATE  OVERHEAD  WITH  RATING 

JUSTIFY  THE  RATING 

ASK  FCR  QUESTIONS  OR  COMMENTS 

NOTE:  IF  THERE  IS  DISAGREEMENT  ON  SOME  OF  THE  RATINGS,  DO  NOT  GO  INTO  A  LENGTHY  DISCUSSION  ABOUT 
THE  DIFFERENCES.  SOME  OF  THE  RATINGS  MAY  BE  JUDGMENT  CALLS  SINCE  THEY  ARE  BASED  ON  LIMITED 
INFORMATION.  CONSIDER  IT  TO  BE  SUFFICIENT  IF  A  PERSON'S  RATING  IS  WITHIN  1  POINT  OF  THE  ASSIGNED 
RATING.  EMPHASIZE  TO  THE  GROUP  THAT  THE  PRIMARY  PURPOSE  OF  THE  EXERCISE  IS  TO  FAMILIARIZE  THEM 
WITH  THE  RATING  FORMS. 

COLLECT  EXERCISE  I.  THEN  SAY: 

NOW  TURN  TO  EXERCISE  II  IN  YOUR  TRAINING  BOOKLET. 


B.  EXERCISE  II  -  DIMENSIONAL  RATING  FORM 


Read  the  following  story  and  use  the  information  in  it  to  complete  a  sample  Dimensional  Rating 
Form  on  Airman  Jones. 

HAND  OUT  EXERCISE  II  -  FLIGHTLINE  DIMENSIONAL  RATING  FORM.  THEN  SAY: 

THIS  RATING  FORM  HAS  TWO  VERSIONS.  ONE  IS  SPECIFICALLY  DESIGNED  FOR  FLIGHTLINE  PERSONNEL;  THE 
OTHER,  FOR  SHOP.  FOR  PRACTICE  PURPOSES,  WE  WILL  USE  ONLY  THE  FLIGHTLINE  FORM.  NOW,  PLEASE  READ 
THE  STORY,  MAKE  YOUR  RATINGS,  AND  THEN  WE  WILL  DISCUSS  THEM. 

EXERCISE  II 

DIMENSIONAL  RATING  FORM 


i 


During  a  recent  Operational  Readiness  Inspection,  Airman  Jones  was  called  upon  to 
troubleshoot  a  starter  system  malfunction  on  a  674  aircraft.  He  was  selected  because  he  can 
complete  most  flightllne  maintenance  tasks  with  some  supervision  (5-3). 


Airman  Jones  required  substantial  assistance  from  his  supervisor  to  diagnose  that  the  starter 
control  valve  was  inoperative  (6-2),  He  removed  and  replaced  the  component  in  the  required 
amount  of  time  and  without  supervision  (2-3).  He  also  inspected  his  work  after  completing  the 
task.  Sgt  Smith  reinspected  the  installation  of  the  starter  control  valve,  finding  only  one 


minor  discrepancy  (3-3).  which  Airman  Jones  quickly  corrected, 
forms,  and  the  aircraft  departed  on  schedule. 


Sgt  Smith  signed  off  the  aircraft 


f 

I 


After  the  aircraft's  departure.  Airman  Jones  completed  the  required  AFTO  Forms  349  and  350. 
He  made  all  the  proper  entries,  but  needed  some  supervision  in  locating  a  couple  of  codes  (1-3). 

Later  in  the  day,  a  request  came  in  from  another  base  to  have  an  engine  shipped  to  them  as 
soon  as  possible.  Airman  Jones  was  assigned  to  perform  a  quality  inspection  on  the  engine  before 
shipment.  He  made  a  thorough  inspection  of  the  engine.  Sgt  Davis  from  quality  control  came  in 
and  made  a  follow-up  inspection  and  noted  only  four  minor  discrepancies  (4-4).  Airman  Jones 
corrected  the  discrepancies,  and  the  engine  was  prepared  for  air  shipment. 


CONDUCT  THE  DISCUSSION  OF  THE  RATINGS  IN  THE  SAME  MANNER  AS  EXERCISE  I,  USING  THE  FOLLOWING 
GUIDELINES: 


jpwwuu  wuuwmww  wwiKwwxffw  bw?  Hj'  vni.w'vjvvvvw'v.Tv; vjrur*r<rvr 


ASK  THE  GROUP  FOR  THEIR  RATINGS 
SHOW  THE  APPROPRIATE  OVERHEAD  WITH  RATING 
JUSTIFY  THE  RATING 
ASK  FOR  QUESTIONS  OR  COMMENTS 
VI.  Conclusion 

Now  that  you  have  become  familiar  with  the  rating  forms  and  have  practiced  making  ratings,  you 
are  ready  to  begin  to  use  the  forms  to  rate  your  performance  or  the  performance  of  your 


APPENDIX  F:  TRAINEE  BOOKLET 


I.  Introduction 

For  the  next  several  hours,  your  time  will  be  spent  focusing  on  the  rating  forms  you  will 
complete  as  part  this  research  project.  You  will  use  several  rating  forms  to  rate  the 
performance  of  your  coworkers  and  yourself.  In  adaition,  your  supervisor  ana  several  coworkers 
will  use  the  same  forms  to  rate  you. 

Before  you  use  any  of  the  rating  forms,  we  are  going  to  talk  about  each  form,  its  purpose,  ana 
how  to  use  each  form  to  effectively  rate  a  coworker  or  yourself.  We  are  also  going  to  discuss 
some  ideas  that  will  help  you  use  the  rating  forms  and  make  the  most  accurate  ratings  possible. 

It  is  essential  to  the  outcome  of  this  project  that  you  be  truthful  and  honest  in  your  ratings. 
The  ratings  will  not  be  seen  by  your  coworkers,  supervisors,  or  anyone  else  connected  with  your 
unit.  The  data  collected  will  be  seen  only  by  Air  Force  Human  Resources  Laboratory  personnel  and 
the  private  contractor  associated  with  this  project.  The  information  you  provide  will  be  coded 
to  assure  anonymity,  and  the  rating  forms  will  subsequently  be  destroyed.  The  ratings  will  be 
used  for  research  purposes  only  and  will  in  no  wav  affect  anyone's  career.  Therefore,  please 
rate  each  person  as  accurately  as  possible. 

II.  Explanation  of  Rating  Scale 

A.  Each  rating  forms  uses  a  5-point  scale  (5  high  -  1  low).  To  help  you  make  more 
accurate  ratings,  the  points  on  the  scale  are  labled,  or  anchored.  You  will  notice  that  these 
anchors  describe  a  certain  level  of  proficiency.  Proficiency  refers  to  how  skilled  a  person  is 
at  performing  various  tasks  on  the  job,  ignoring  interpersonal  factors  (willingness  to  work, 
cooperating  with  others)  or  situational  factors  (lack  of  tools  or  parts,  weather  conditions). 

B.  One  of  the  rating  forms  you  will  be  using  is  not  technical  in  its  orientation.  The 
scale  anchors  of  this  form  refer  to  various  levels  of  performance  rather  than  levels  of 
proficiency.  The  rating  form  will  examine  not  only  technical  ability,  but  also  other  factors 
that  contribute  to  a  mechanic's  performance  on  the  job. 

C.  Three  of  the  rating  forms  will  Include  short  paragraphs  that  describe  behavior  typical 
to  each  level  on  the  scale.  These  behavioral  examples  will  aid  you  in  making  your  ratings. 

III.  Explanation  of  Rating  Forms 

A.  Global  Rating  Form 

This  rating  form  has  two  Items.  The  first  item  asks  for  a  rating  of  a  person's  overall 

technical  proficiency.  The  second  item  asks  for  a  rating  of  a  person's  overall 

social/interpersonal  proficiency.  This  form  utilizes  behaviorlal  examples  to  aid  you  in  making 
your  ratings. 

B.  Dimensional  Rating  Form 

The  purpose  of  this  rating  form  is  to  evaluate  a  mechanic's  proficiency  on  a  number  of 
important  job  areas  or  dimensions.  Again,  the  behavioral  examples  will  serve  as  a  guide  in 
determining  your  ratings. 

C.  Task  Rating  Form 

The  purpose  of  this  rating  form  is  to  evaluate  proficiency  on  a  variety  of  tasks 

critical  to  first-term  Jet  Engine  Mechanics.  This  form  contains  tasks  common  to  the  J-57,  J-79, 
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TF-33  engines  and  to  the  flightline  and  shop  areas.  It  also  contains  items  unique  to  the  engine 
you  work  on  and  your  functional  area. 

0.  Air  Force-Wide  Rating  Form 

This  rating  form  does  not  evaluate  technical  job  skill.  It  rates  a  person  on  elements 
important  to  overall  success  in  the  Air  Force.  This  is  the  rating  form  mentioned  previously  that 
uses  performance  rather  than  proficiency  as  a  basis  for  rating  scale  anchors.  Again,  you  will 
utilize  behavioral  examples  in  making  your  ratings. 

E.  Supplemental  Questionnaires 

You  will  notice  that  there  are  several  other  questionnaires  included  in  the  booklet. 
On  pages  1-2  is  a  General  Background  questionnaire  which  asks  for  information  on  yourself  and  the 
person  you  are  rating.  The  Rating  Form  Questionnaire,  found  on  pages  25-26,  asks  you  to  give  an 
evaluation  of  the  rating  forms.  The  final  item  in  the  booklet  is  the  Task  Experience  Ratings 
Questionnaire,  which  requires  you  to  provide  information  on  how  much  experience  you've  had  on 

various  tasks.  You  will  complete  this  questionnaire  only  on  yourself,  not  on  your  peers, 

supervisors,  or  subordinates. 

IV.  Tips  on  Making  Accurate  Ratings 

A.  General  Information 

The  most  Important  thing  to  remember  when  making  your  ratings  is  to  focus  your  attention 
only  on  the  person  you  are  rating,  and  only  on  the  person's  ability  to  perform.  Avoid  comparing 
the  person  with  coworkers  you've  rated  previously  or  those  you  will  subsequently  evaluate. 

Remember  that  even  though  you  and  the  people  you  rate  are  similar  because  you  are  all 
Jet  Engine  Mechanics,  each  person  has  a  unique  set  of  strengths  and  weaknesses.  Also,  be  aware 
that  poor  or  outstanding  performance  in  one  area  does  not  dictate  the  quality  of  performance  in 
other  areas.  Rate  the  person  according  to  the  individual's  ability  to  perform  on  the  job,  and 

focus  on  observable  behavior.  Finally,  do  not  be  afraid  to  use  the  entire  range  of  the  scale 

when  appropriate.  Your  honesty  will  serve  to  ensure  the  accuracy  of  your  ratings. 

B.  Exercise 

The  following  Is  a  conversation  between  several  engine  crew  chiefs.  Read  it  and  see  if 
you  can  identify  factors  that  might  contribute  to  Inaccurate  ratings. 

Andrews:  Here  we  are  again  at  our  weekly  gripe  session. 

Baker:  Yeah.  Seems  like  we  never  run  out  of  problems  to  complain  about. 

Cullen:  I  don't  know  why  you  guys  have  so  many  problems  on  your  crew. 

Baker:  Because  we  don't  have  the  cream  of  the  crop  that  you  have. 

Cullen:  You've  got  the  same  kind  of  guys  working  for  you  that  I  have.  Just  think  about 
It.  All  the  airmen  take  tests  to  Identify  who  has  the  aptitude  for  what.  Right? 

Andrews:  Right. 


Cullen:  OK.  So  you've  got  a  bunch  of  guys  with  mechanical  aptitude.  These  guys  all 
receive  the  same  training  at  tech  school  to  learn  to  be  competent  mechanics.  They 
all  come  out  of  tech  school  with  the  same  training.  That's  the  purpose  of  tech 


school.  So  there  shouldn't  be  anyone  on  your  crew  who  is  more  qualified  to  do  the 
job  than  someone  else.  Granted,  if  you  take  any  one  of  the  airmen  and  compare  him 
to  the  average  guy  on  the  street,  sure,  he  is  going  to  look  like  a  mechanical 
genius.  But  among  his  fellow  crew  members,  he's  just  another  average  guy  like  all 
the  rest. 

Andrews:  I  disagree.  On  my  crew.  I've  got  a  couple  of  guys  who  are  just  outstanding.  Next 
to  tnese  guys,  the  rest  of  my  airmen  are  way  below  average. 

Baker:  I  know  what  you  mean.  I  received  a  new  recruit  2  weeks  ago.  You  wouldn't  believe 

how  badly  he  has  performed  in  the  short  time  he's  been  here.  I  guess  I've  got  a 
real  loser  on  my  hands.  He'd  be  right  at  home  with  your  guys,  Andrews. 

Oavis:  I  guess  I  sort  of  understand  what  you  are  talking  about,  but  on  the  other  side  of 

the  coin.  The  last  recruit  that  came  onto  my  crew  was  fresh  out  of  tech  school. 

He  had  been  here  only  a  week  or  so  and  I  gave  him  AFTO  Form  349  to  complete.  Hell, 
he  completed  the  form  without  even  asking  a  question  and  used  the  06  Code  Manual 
like  a  pro.  I  figure  I  have  a  super  mechanic  in  this  guy.  I  mean,  if  he  can  do 
that,  he  can  do  just  about  anything  I  assign  him  to  do. 

Andrews:  You  know,  Cullen,  I've  heard  how  difficult  you  can  be  when  you  rate  your  airmen. 

I've  heard  some  of  your  people  complain  that  no  matter  how  hard  they  try,  you  won't 

give  them  more  than  an  average  rating.  They  say  they  feel  like  they're  banging 

their  heads  against  a  brick  wall. 

Cullen:  I  have  a  pretty  good  idea  who  you're  talking  about.  If  they  think  it's  hard  to  get 
a  good  rating  out  of  me  now,  just  wait  until  I  rate  them  next  time.  They'll  find 
out  how  hard  a  brick  wall  really  Is, 

Andrews:  I  guess  I  don't  think  like  you  guys  at  all.  I  try  to  be  a  friend  to  my  airmen. 

Some  of  them  are  away  from  home  for  the  first  time  and  I  don't  like  to  make  it  any 
rougher  on  them  than  I  need  to.  I  try  to  give  my  airmen  the  benefit  of  the  doubt 

when  I'm  rating  them.  You'd  be  surprised  what  you  learn  about  your  airmen  if  you 

try  to  be  their  friend.  For  Instance,  there's  a  guy  on  my  crew  now  who  reminds  me 

of  myself  when  I  was  his  age.  He's  from  a  small  town  and  joined  the  Air  Force  to 

get  away  from  that  atmosphere  and  see  a  little  of  the  world.  I  can  really  identify 

with  him. 

Oavis:  Well,  I  don't  know  about  being  buddy-buddy  with  my  airmen,  but  I  do  know  that  right 

now,  everyone  In  n\y  unit  is  doing  a  good  job  as  far  as  I'm  concerned. 

Andrews:  Why  Is  that? 

Oavis:  Well,  I'm  up  for  promotion  In  a  couple  of  months.  And  let's  face  it,  the  better  iqy 

airmen  look  on  their  ratings,  the  better  I  look  as  a  supervisor.  And  the  better  I 
look,  the  better  my  chances  are  for  promotion'. 

V.  Practice  Exercises  for  Rating  Forms 

Now  you  will  have  the  opportunity  to  practice  using  two  of  the  rating  forms  that  have  been 

discussed,  the  Air  Force-Wide  and  the  Olmenslonal.  The  purpose  of  these  exercises  is  to  help  you 

become  familiar  and  comfortable  with  the  forms  and  to  address  any  questions  or  concerns  you  might 
have  regarding  their  use. 
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A.  Exercise  I  -  Air  Force-Wide  Rating  Form 

Read  the  following  story  and  use  the  information  in  it  to  complete  a  sample  Air  Force-Wide  Rating 
Form  on  Airman  Martin. 


EXERCISE  I 


AIR  FORCE-WIDE  RATING  FORM 

Airman  Martin,  an  engine  crew  chief,  was  in  the  process  of  reassembling  his  engine  when  the 
shop  chief  informed  him  that  the  engine  had  to  be  ready  for  final  inspection  by  Friday. 

Airman  Martin  said  that  there  would  be  no  problem  meeting  that  deadline,  and  he  quickly  took 
charge  of  the  situation,  gaining  the  support  of  his  subordinates.  Using  his  technical  knowledge 
and  skills,  he  finished  the  assembly  late  Thursday  afternoon  but  required  supervision  during  the 
buildup  of  the  next  section. 

Prior  to  leaving.  It  was  discovered  that  a  tool  was  missing.  Being  a  truthful  individual, 
and  knowing  the  Importance  of  the  situation.  Airman  Martin  informed  the  shop  chief  immediately. 
Staying  calm  and  In  control,  he  volunteered,  without  hesitation,  to  stay  until  the  tool  was  found. 

The  next  morning.  Airman  Martin  was  informed  to  report  to  the  Branch  Chief  at  1300.  At 
1300,  he  reported  to  the  Branch  Office.  The  Branch  Chief  called  him  in  and  commended  him  on  a 
job  well  done  and  on  his  self-control  in  the  situation  of  the  missing  tool.  Also,  the  Branch 
Chief  said  his  appearance  and  military  btsrlng  were  outstanding  and  to  keep  up  the  good  work. 

After  getting  off  duty  later  that  day.  Airman  Martin  went  to  the  Tech  Order  room  to  check 
out  some  T.O.s  to  study  the  problem  on  the  engine  he  was  getting  next. 

B.  Exercise  II  -  Oimensional  Rating  Form 

Read  the  following  story  and  use  the  Information  In  it  to  complete  a  sample  Dimensional  Rating 
Form  on  Airman  Jones. 


EXERCISE  II 

DIMENSIONAL  RATING  FORM 


During  a  recent  Operational  Readiness  Inspection,  Airman  Jones  was  called  upon  to 
troubleshoot  a  starter  system  malfunction  on  a  674  aircraft.  He  was  selected  because  he  can 
complete  most  flightline  maintenance  tasks  with  some  supervision. 

Airman  Jones  required  substantial  assistance  from  his  supervisor  to  diagnose  that  the 
starter  control  valve  was  Inoperative.  He  removed  and  replaced  the  component  in  the  required 
amount  of  time  and  without  supervision.  He  also  Inspected  his  work  after  completing  the  task. 
Sgt  Smith  reinspected  the  Installation  of  the  starter  control  valve,  finding  only  one  minor 
discrepancy,  which  A-irman  Jones  quickly  corrected.  Sgt  Smith  signed  off  the  aircraft  forms,  and 
the  aircraft  departed  on  schedule. 

After  the  aircraft's  departure.  Airman  Jones  completed  the  required  AFTO  Forms  349  and  350. 
He  made  all  the  proper  entries,  but  needed  some  supervision  in  locating  a  couple  of  codes. 
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Now  that  you  have  become  familiar  with  the  rating  forms  and  have  practiced  making  ratings,  you 
are  ready  to  begin  to  use  the  forms  to  rate  your  performance  or  the  performance  of  your 
co-workers  or  subordinates.  Try  to  make  the  most  accurate  ratings  possible,  keeping  in  mind  the 
tips  that  were  discussed  during  this  training  session.  Remember  that  the  information  collected 
on  the  rating  forms  will  be  used  for  research  purposes  only.  It  will  not  go  into  anyone's  record 
or  be  seen  by  persons  other  than  research  personnel.  Also,  please  read  all  instructions 
carefully. 


*.*  »-«'»  t  t  *1.  i».  i».  4*»  i»«  i».  n.  |*a  it,  it,  iU  i’»‘rifc  j|t  «k_ >kt.**l.  ft'. 


SAMPLE  RATING  FORM  BOOKLET 
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GENERAL  INSTRUCTIONS 


You  are  being  asxed  to  participate  in  a  research  project  aimed  at  determining  how  useful 
different  types  of  rating  forms  are  for  evaluating  a  first-term  airman's  job  performance. 
Throughout  the  project,  first-term  Jet  Engine  Mechanics  will  be  asked  to  complete  these  rating 
forms  on  themselves  and  some  of  their  coworkers.  In  addition,  NCOs  who  are  direct  supervisors  of 
the  airmen  will  be  asked  to  provide  evaluations. 


The  information  provided  from  the  rating  forms  will  be  used  solely  for  research  purposes.  It 
will  not  be  seen  by  other  airmen  or  anyone  else  connected  with  your  unit.  In  fact,  we  are  not 
concerned  with  the  ratings  an  individual  receives  or  with  how  an  individual  rates  others.  The 


purpose  of  this  project  is  to  answer  the  question,  "Which  rating  forms  are  most  useful  for  making 
accurate  evaluations?"  These  ratings  will  in  no  wav  be  associated  with  you  or  anyone  involved; 
so,  please  be  totally  honest  in  your  evaluations. 


Information  will  be  collected  from  many  Jet  Engine  Mechanics  at  a  number  of  Air  Force  bases 
across  the  country.  These  data  will  help  us  decide  which  rating  forms  improve  the  chances  of 
getting  accurate  evaluations.  In  addition,  information  will  be  gathered  about  recruit  qualities 
that  lead  to  success  in  the  Air  Force.  Because  subsequent  decisions  regarding  the  quality  of  the 
various  rating  forms  will  be  based  on  information  provided  by  participants  such  as  yourself,  it 
is  essential  that  you  take  whatever  time  necessary  to  give  the  most  accurate  ratings  possible. 


This  rating  form  booklet  contains  four  main  sections.  First,  you  will  be  asked  to  complete  a 
general  background  questionnaire  that  asks  general  questions  about  yourself,  your  jet  engine 
experience,  and  your  time  in  service.  The  second  section  contains  four  (4)  different  rating 
forms.  Each  rating  form  was  developed  through  extensive  contact  with  Jet  Engine  Mechanics. 
These  job  experts  both  developed  and  reviewed  these  forms;  therefore,  we  believe  the  content  of 
the  forms  accurately  reflects  the  job  of  a  Jet  Engine  Mechanic.  These  four  ratings  forms  will 
include  a  2-item  global  rating  form,  a  more  specific  dimensional  rating  form,  a  detailed, 
task-specific  form  and  an  Air  Force-wide  rating  form. 


All  ratings  are  made  on  a  scale  with  five  points  (5-high,  1-Iow).  A  rating  of  5  indicates  that 
you  always  exceed  the  acceptable  level  of  proficiency;  and  a  rating  of  1,  that  you  never  meet  the 
acceptable  level  of  proficiency.  Specific  Instructions  for  completing  each  rating  form  are 
included  as  part  of  the  form. 


The  third  section.  Rating  Form  Questionnaire,  asks  for  your  reactions  concerning  the  usefulness 
of  the  four  different  rating  forms.  The  fourth  section  asks  you  to  describe  how  much  experience 
you  have  had  performing  tasks  representative  of  what  a  first-term  Jet  Engine  Mechanic  does  on  the 
job.  Please  complete  all  four  sections  of  this  booklet. 


Thank  you  very  much  for  giving  this  your  careful  attention. 
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APPENOIX  G:  WTPT  TEST  ADMINISTRATOR  TRAINING  AGENDA 


Date: 
PI  ace: 


2-12  September  1986 
328X0  -  Room  218,  Bldg  578 
272X0  -  Room  175,  Bldg  578 
492X1  -  Room  1R,  Bldg  180 


Day  1  -  Tuesday.  2  Sep  86 


AFHRL  Introduction 
JPM  Project  Overview 

Baseline  Scoring  Exercise  (Proctors  not  present) 

Distribute  Test  Administrator  Materials  (Proctors  not  present) 


♦Proctors  and  Test  Administrators  for  all  specialties  will  receive 
Introduction  and  Overview  in  Room  218. 


Day  2  -  Wednesday.  3  Sep  86 

Test  Administrator  Requirements  (4  hours) 

WTPT  Administrator's  Manual 
General  Logistics 
Special  Requirements/Other 

Hands-On  Instrument  Training  (4  hours) 

Look  at  Instruments  and  Begin  to  Oiscuss  Overall  Performance  Ratings 
Phase  I  Hands-On  Task  Training 


Day  3  -  Thursday.  4  Sep  86 

Hands-On  Instrument  Training 

Continue  Phase  I  Hands-On  Task  Training  (Proctors  not  present) 
Phase  II  Hands-On  Task  Training  (Proctors  not  present) 


Day  4  -  Friday,  5  Sep  86 

Interview  Training 

Lecture  on  Interviewing 
Modeling  Exercise 
Role-Playing 

Phase  I  Interview  Task  Training 


Day  5  -  Monday,  8  Sep  86 

Phase  II  Interview  Task  Training  (4  hours,  Proctors  not  present) 
WTPT  Answer  Sheet  Training  (4  hours,  Proctors  present) 


Post-Training  Test  (4  hours,  Proctors  not  present) 

Train  on  Control  Group  Tasks  (4  hours,  Proctors  not  present) 
Wrap  up  Overall  Performance  Ratings 


1 


Day  7  -  Wednesday,  10  Sep  86 


Team  Development 

Role  Clarification 
In-Depth  Logistics 

"A  Oay  in  the  Life  of  a  Test  Administrator" 


Day  8  -  Thursday,  11  Sep  86 


Proctors'  Rater  Training  Dress  Rehearsal  -  Test  Administrators  and 
AFHRL  Personnel  Observe 


Day  9  -  Friday,  12  Sep  86 


For  492X1  and  (possibly)  272X0  -  Criticality/Importance  Ratings 

For  328X0  -  Time  to  Proficiency 

Space  Perception  Test 

Material  Assembly/Last-Minute  Details 

Pep  Rally 
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APPENDIX  H;  NOTES  ON  COLLECTION/DISCUSSION  OF 
INTERRATER  RELIABILITY  INFORMATION 


I.  Introduction:  Three  major  components. 

A.  Shadow  scoring. 

B.  Field  Test  Administrator  (TA)  Facilitation. 

C.  Calibration. 

All  involve  proctors  directing  test  administrator  (TA)  training--the  focus  is  to  move  TAs  toward 
consistency/accuracy  of  incumbent  observation/scoring. 

II.  Shadow  Scoring. 

A.  Objective:  to  collect  information  on  how  TAs  score  individuals  during  field  test. 

B.  Procedures. 

1.  At  each  base  have  TAs  score  several  incumbents  together. 

(a)  Number  of  incumbents  scored  this  way  depends  on  total  number  of  incumbents  to 
be  tested  and  projected  length  of  stay. 

(b)  Optimal  number  to  be  tested  3-5. 

(1)  Key  constraint--can  total  WTPT  testing  be  completed  in  alloted  time? 

(2)  If  extra  shadow  scoring  forces  stay  at  base  to  extend  over  weekend,  or 
will  Interfere  with  projected  arrival  at  next  base,  reduce  number  of 
incumbents  to  be  shadow  scored. 

2.  Proctor  should  estimate  (prior  to  testing)  required  testing  time/logistical 
constraints,  and  make  decision  on  number  of  incumbents  to  be  shadow  scored.  If 
questions  arise,  contact  UES  or  AFHRL  for  input.) 

3.  For  each  shadow-scored  Incumbent,  proctor  should  designate  one  TA  as  primary  (TA  who 
will  Interact  with  incumbent);  primary  role  should  be  rotateo  between  TAs  for  each 
incumbent. 

4.  Proctor  should  ensure  that  TAs  score  independently  (without  interaction  or  knowledge 
of  how  other  TA  is  scoring);  independent  scoring  is  critical  and  should  be  emphasized  by 
proctor. 

5.  At  end  of  day,  meet  with  TAs  and  go  over  booklets. 

(a)  Talk  about  differences  found  and  why. 

(b)  Try  to  reach  agreement  about  what  the  score  (go/no  go)  on  each  step  should 
have  been. 

(c)  Talk  about  overall  performance  score  and  why  rating  was  given. 

(1)  Once  again  try  to  reach  consensus  about  what  rating  was  correct. 

(2)  It  may  be  useful  to  revisit  performance  criteria  to  be  used  (as  discussed 
In  TA  training  sessions  at  AFHRL). 

6.  DO  NOT  CHANGE  ANY  SCORES  as  a  result  of  discussions;  original  scores  must  be  entered 
on  opscan  forms. 
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7.  Repeat  this  procedure  each  day  until  shadow  scoring  ends. 


C.  Concluding  Comments. 


1.  Interrater  reliability  observations  from  each  trip  need  to  be  compiled  and  used  upon 
return  to  San  Antonio  and/or  at  next  base. 


2.  Information  should  be  analyzed  and  discussed  with  TAs  in  terms  of: 

(a)  Identifying  TA  problems. 

(b)  Specific  tasks  identified  that  need  additional  videotape  training/discussion. 

(c)  General  thoughts/comnents  concerning  procedures  to  follow  in  future  test 
administrations. 


3.  All  observations/ information  gathered  should  be  documented. 


III.  Field  Test  Administration  Facilitation. 


A.  Objective  is  to  observe  test  administration;  save  problems/peculiarities/issues  for 
later  discussions  with  TAs. 


B.  Procedures. 


1.  During  test  administration,  proctors  should  monitor  scoring  by  TAs,  noting: 

(a)  Discrepancies  in  marking  booklets. 

(b)  Excessive  interaction  with  incumbent/improper  probing  during  interview. 

(c)  Problems  with  building  rapport,  deviations  from  proper  procedures,  transition 
between  tasks,  etc. 


2.  At  appropriate  time  (end  of  day,  end  of  incumbent  testing,  end  of  task),  discuss 
problems  noted. 

(a)  Seriousness  of  problems  should  dictate  when  discussion  between  proctor  and  TA 


(b)  At  end  of  each  day,  proctor  should  meet  with  TAs  to  discuss 
activities/observations  of  the  day. 


C.  Concluding  Comments. 


1.  Prior  to  returning  to  San  Antonio  (or  arrival  at  next  base),  information  gathered  in 
the  field  should  be  collated  and  documented. 


2.  As  problems/concerns  arise,  proctors  may  want  to  communicate  with  each  other,  or 
call  UES  or  AFHRl  If  additional  clarification  is  needed. 


IV.  Test  Administrator  Calibration. 


A.  Objective:  Using  videotapes  and  discussion,  proctors  should  review  task  scoring  and 
calibrate/retrain  TAs  to  previous  levels  of  accuracy/consistency. 


B.  Procedures. 


1.  When  teams  return  to  San  Antonio,  workshop  should  be  held  (a  maximum  of  5  days  will 
be  available  for  this  activity,  but  hopefully  3  days  will  be  sufficient). 


2.  Prior  to  workshop,  proctors  for  a  specialty  should  meet  and  discuss  field 
observations,  problem  tasks,  etc.  for  discussion  in  workshop. 
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3.  Topics  of  discussion: 


(a)  Observations  in  the  field. 

(o)  Review  of  all  videotapes  in  same  manner  as  used  in  original  workshop. 

(1)  Each  task  observed/scored. 

(2)  Scoring  discussed  after  each  task;  aim  is  to  ensure  TA 
accuracy/consistency  of  observing  and  scoring. 

(3)  Overall  performance  score  should  also  be  discussed  in  the  same  manner; 
criteria  used  to  arrive  at  decision  should  be  emphasized. 

(c)  Specific  problem  tasks  noted  in  field  should  be  emphasized,  and  possibly 
observed/scored  several  times. 

(d)  Some  tasks  have  been  videotaped  correctly  and  incorrectly  (several  versions), 
and  consequently  provide  additional  tapes  to  be  observed/scored/discussed. 

4.  Scored  booklets  should  be  retained  (with  original  scores  recorded)  for  later 
analysis. 

5.  Proctor  should  be  satisfied  that  problems  have  been  corrected  before  ending  workshop. 
C.  Concluding  Comments. 

1.  After  first  return  to  San  Antonio  for  team;  subsequent  recalibration  may  occur  in 
the  field  prior  to  data  collection  if  videotape  equipment/facilities  can  be  obtained. 

2.  During  this  workshop,  additional  areas  to  work  on  may  include: 

(a)  Interviewing  skills  (probing  techniques,  etc.) 

(b)  Logistical  Issues. 


APPENDIX  I:  RATER  TRAINING  EXERCISES  I  AND  II  FOR  AFS  491X2 


•NOTE  TO  ADMINISTRATORS: 

The  underlined  statements  in  Exercise  I  and  II  are  justifications  for  ratings.  The  parenthetical 
notation  following  each  statement  refers  to  the  dimension/performance  factor  number  and  its 
appropriate  rating. 


EXERCISE  I:  AIR  FORCE-WIDE  RATING  FORM 

Martin,  a  first-term  radio  operator,  was  in  the  process  of  configuring  his  radio  equipment 
when  the  shift  supervisor  informed  him  thare  was  a  priorty  phone  patch  coming  through  at  any 
moment. 

Martin  said  there  would  be  no  problem  in  handling  the  patch;  and  he  quickly  took  charge  of 
the  situation,  gaining  the  support  of  his  subordinates  (PF5-5).  Using  technical  knowledge  and 
skills,  he  completed  configuring  his  radio,  but  required  supervision  to  engage  the  phone  patch 
equipment  (PF1-3). 

Prior  to  leaving  for  the  day,  it  was  discovered  that  a  call  sign  list  was  missing.  Being  a 
truthful  individual,  and  knowing  the  importance  of  the  situation.  Martin  informed  the  shift 
supervisor  immediately  (PF4-5).  Staying  calm  and  in  control,  he  volunteered  without  hesitation 
to  stay  until  the  call  sign  list  was  found  (PF2-5). 

The  next  morning,  Martin  was  Instructed  to  report  to  the  Director  of  Operations  at  1300.  At 
1300,  he  reported  to  the  DO  office  (PF3-4).  The  DO  called  him  in  and  commended  him  on  a  job  well 
done  and  on  his  self-control  in  the  situation  of  the  missing  call  list  (PF8-5).  Also,  the  DO 
said  his  appearance  and  military  bearing  were  outstanding  (PF6-5)  and  to  keep  up  the  good  work. 

After  getting  off  duty  later  that  day,  Martin  went  to  the  publication  room  to  check  out  some 
T.O.s  to  study  the  radio  equipment  he  must  qualify  on  next  (PF7-4). 


EXERCISE  II:  DIMENSIONAL  RATING  FORM 

Ouring  a  recent  Operational  Readiness  Inspection,  Jones  was  called  upon  to  configure  the  Scope 
Signal  III  and  operate  a  radio.  He  was  selected  because  he  can  complete  most  radio  configuration 
tasks  with  only  some  supervision  (01-3).  Jones  required  substantial  assistance  from  his 
supervisor  when  coordinating  with  HQ  SAC  to  handle  an  aircraft  emergency  (D2-2).  He  received  and 
transcribed  several  messages  with  only  minor  errors  (D5-4),  He  also  verified  the  incoming 
messages  using  a  coil  sign  list  without  supervision  (03-5).  Finally,  he  completed  the  required 
AF  1020.  He  made  all  the  proper  entries,  but  needed  some  supervision  in  completing  the  remarks 
column  (04-3). 


